2024-03-29T10:35:27.842 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:35:27.842 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/orig.config.yaml 2024-03-29T10:35:27.877 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:35:27.960 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622655' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1XYCHPDgbbRvcQkzvM0nSiBLU6dxAKguv4WJ+e8cGFf0JzLEoLa3QcVarrIPCZovi3uCVThv/ja51YOi9SH7k= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJI+GBK6tFBCw/9r9EfQ2evtADR04/h6CTSao5wyoWR+iyOenbhJP8D8WKEC+5hpebwKZf8C1TUhmL9pYHsv20c= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:35:27.960 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:35:27.961 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:35:27.961 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:35:27.963 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:35:27.988 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:35:27.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:35:27.988 INFO:teuthology.packaging:ref: None 2024-03-29T10:35:27.988 INFO:teuthology.packaging:tag: None 2024-03-29T10:35:27.988 INFO:teuthology.packaging:branch: main 2024-03-29T10:35:27.988 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:35:27.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:35:28.359 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:35:28.361 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:35:28.366 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:35:28.366 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:35:28.372 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:35:28.385 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:35:28.391 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:35:28.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.513095', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1XYCHPDgbbRvcQkzvM0nSiBLU6dxAKguv4WJ+e8cGFf0JzLEoLa3QcVarrIPCZovi3uCVThv/ja51YOi9SH7k='} 2024-03-29T10:35:28.432 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.511840', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJI+GBK6tFBCw/9r9EfQ2evtADR04/h6CTSao5wyoWR+iyOenbhJP8D8WKEC+5hpebwKZf8C1TUhmL9pYHsv20c='} 2024-03-29T10:35:28.432 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:35:28.438 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:35:28.439 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:35:28.439 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:35:28.510 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f09839cc0d0>, signals=[15]) 2024-03-29T10:35:28.511 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:35:28.517 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:35:28.518 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:35:28.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:28.628 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:35:28.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:28.719 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:35:28.731 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-03-29T10:35:28.749 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-03-29T10:35:28.750 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-29T10:35:28.808 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-29T10:35:28.808 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-03-29T10:35:28.808 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-29T10:35:28.808 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:35:28.808 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:35:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:35:28.810 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-29T10:35:28.836 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-03-29T10:35:28.856 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-03-29T10:35:28.857 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-03-29T10:35:28.913 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:35:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:35:28.915 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:35:28.915 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:35:28.915 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:35:28.915 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-03-29T10:35:28.936 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:35:28.954 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:35:28.968 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:35:28.968 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:35:28.971 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:35:28.988 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:35:29.024 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:35:29.024 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:35:29.033 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:35:29.060 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:35:29.126 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:35:29.149 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:35:29.149 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:35:29.150 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:35:29.150 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:35:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:35:29.167 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-03-29T10:35:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:35:29.168 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-29T10:35:29.224 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:35:29.224 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-361.el9.x86_64 2024-03-29T10:35:29.224 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-03-29T10:35:29.226 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:35:29.227 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-03-29T10:35:29.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-03-29T10:35:31.690 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-03-29T10:35:32.484 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 4.0 MB/s | 6.7 MB 00:01 2024-03-29T10:35:33.157 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-29T10:35:33.975 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-29T10:35:35.799 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-29T10:35:38.075 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-29T10:35:42.692 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.1 MB 00:00 2024-03-29T10:35:42.972 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 901 kB/s | 8.0 MB 00:09 2024-03-29T10:35:44.109 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-29T10:35:45.597 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T10:35:47.552 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-03-29T10:35:52.761 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T10:35:55.209 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:35:55.209 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:35:55.310 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:35:55.316 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-03-29T10:35:55.317 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:35:55.318 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 76 M 2024-03-29T10:35:55.318 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-03-29T10:35:55.318 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:35:55.620 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 636 kB/s | 192 kB 00:00 2024-03-29T10:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-03-29T10:35:57.412 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:35:57.955 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 712 kB/s | 6.1 MB 00:08 2024-03-29T10:35:58.070 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 12 MB/s | 34 MB 00:02 2024-03-29T10:35:58.073 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:58.074 INFO:teuthology.orchestra.run.smithi064.stdout:Total 28 MB/s | 76 MB 00:02 2024-03-29T10:35:58.074 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:35:58.531 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:35:58.531 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:35:59.407 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:35:59.407 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:35:59.814 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-03-29T10:36:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:36:02.232 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-29T10:36:02.477 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:02.562 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:05.579 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:07.215 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-29T10:36:09.414 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:36:09.414 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:36:09.515 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:36:09.520 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:36:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-03-29T10:36:09.522 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:36:09.522 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 76 M 2024-03-29T10:36:09.522 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 123 M 2024-03-29T10:36:09.522 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:36:09.666 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:36:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:10.720 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:11.225 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 8.9 MB/s | 15 MB 00:01 2024-03-29T10:36:12.766 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 10 MB/s | 34 MB 00:03 2024-03-29T10:36:13.631 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 7.0 MB/s | 28 MB 00:03 2024-03-29T10:36:13.634 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:13.634 INFO:teuthology.orchestra.run.smithi174.stdout:Total 19 MB/s | 76 MB 00:04 2024-03-29T10:36:13.635 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:36:14.001 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:36:14.002 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:36:14.817 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:36:14.817 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:36:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:17.123 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:36:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:20.501 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:20.718 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:25.414 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:25.575 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:30.282 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:46.938 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:46.938 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:36:46.938 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:36:46.984 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:36:47.901 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:36:48.218 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:48.335 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.335 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.335 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:36:48.335 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:36:48.335 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:36:48.335 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:36:48.336 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:36:48.418 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:48.418 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-03-29T10:36:48.465 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-03-29T10:36:48.465 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-03-29T10:36:48.466 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:36:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:36:48.468 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:36:48.468 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:36:49.056 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-03-29T10:36:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:36:50.512 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-03-29T10:36:50.514 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-29T10:36:50.529 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.KatTLOMopD 2024-03-29T10:36:50.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KatTLOMopD 2024-03-29T10:36:50.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.KatTLOMopD 2024-03-29T10:36:50.747 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.KatTLOMopD is 6KB 2024-03-29T10:36:50.795 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.KatTLOMopD 2024-03-29T10:36:50.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:36:50.888 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:36:50.888 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:36:50.888 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:36:50.888 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:36:50.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:36:51.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-03-29T10:36:59.304 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:59.305 INFO:teuthology.orchestra.run.smithi174.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:36:59.305 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:36:59.348 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:59.716 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:59.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:59.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:59.717 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:37:00.100 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:37:00.101 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:37:00.101 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:37:00.400 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:37:00.496 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:37:00.496 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:37:00.496 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:37:00.496 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:37:00.496 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:37:00.496 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:37:00.496 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:37:00.497 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:37:00.576 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:37:00.576 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:37:00.576 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2024-03-29T10:37:00.618 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2024-03-29T10:37:00.618 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2024-03-29T10:37:00.618 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.06 2024-03-29T10:37:00.618 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 68.el9 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Group : Unspecified 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 8264254 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : builder@centos.org 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2024-03-29T10:37:00.619 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:37:00.620 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:37:00.620 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:37:00.620 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:37:00.620 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2024-03-29T10:37:00.620 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:37:00.620 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:37:00.621 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:37:00.621 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:37:00.621 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:37:01.259 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-03-29T10:37:02.629 INFO:teuthology.orchestra.run.smithi174.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:37:02.660 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-03-29T10:37:02.662 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-03-29T10:37:02.677 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.yLfC8fV04E 2024-03-29T10:37:02.678 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yLfC8fV04E 2024-03-29T10:37:02.747 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.yLfC8fV04E 2024-03-29T10:37:02.883 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.yLfC8fV04E is 6KB 2024-03-29T10:37:02.931 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.yLfC8fV04E 2024-03-29T10:37:02.945 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:37:03.018 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:37:03.018 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:37:03.018 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:37:03.018 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:37:03.019 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:37:03.181 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-03-29T10:37:21.054 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:37:21.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T10:37:21.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:37:33.194 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:37:33.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-29T10:37:33.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:07.647 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-03-29T10:38:16.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-29T10:38:16.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:21.062 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:38:30.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T10:38:30.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:31.514 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-29T10:38:32.008 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T10:38:32.008 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:38:32.008 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-29T10:38:32.022 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:38:32.022 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:38:32.022 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:38:32.022 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:38:33.024 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:38:33.024 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:38:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-03-29T10:38:33.114 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:35.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-03-29T10:38:47.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-29T10:38:47.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:47.438 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-29T10:38:47.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-29T10:38:47.894 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:38:47.894 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-03-29T10:38:47.908 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:38:47.908 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:38:47.908 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:38:47.908 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:38:48.909 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:38:48.909 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:38:49.027 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-03-29T10:38:49.077 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:49.077 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:38:49.085 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:38:49.085 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:38:49.088 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:38:49.103 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:38:49.114 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:38:49.129 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:38:49.129 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:38:49.145 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:38:49.212 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:38:49.224 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:38:49.225 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:38:49.227 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:38:49.257 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:49.266 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:49.272 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:49.278 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:49.280 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:38:49.289 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:38:49.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:38:49.316 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:38:49.349 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:38:49.360 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:38:49.361 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:38:49.386 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:38:49.405 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:49.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:49.531 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:49.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:49.656 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:38:49.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:38:49.724 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:49.773 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:49.828 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:49.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:49.952 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:38:49.952 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:38:50.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-03-29T10:38:50.026 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-03-29T10:38:50.091 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:38:50.169 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:38:50.503 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:38:50.512 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:38:50.512 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:38:50.527 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:38:50.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-03-29T10:38:50.597 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-03-29T10:38:50.598 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-03-29T10:38:50.683 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-03-29T10:38:50.688 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:38:50.688 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-03-29T10:38:50.718 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-03-29T10:38:50.719 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-03-29T10:38:50.754 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-03-29T10:38:50.754 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:38:50.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:38:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:38:50.788 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:38:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:38:50.844 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:38:50.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:38:50.878 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:38:50.932 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:38:50.942 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:38:50.965 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:38:50.967 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:43:04.570 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-03-29T10:43:04.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T10:43:04.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:04.664 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-29T10:43:04.740 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-29T10:43:04.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-29T10:43:04.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:04.830 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-29T10:43:04.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-29T10:43:04.908 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:43:04.919 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:43:04.919 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:43:04.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:43:04.921 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:43:04.922 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:43:04.950 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:43:04.967 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:43:04.979 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:43:04.995 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:43:04.996 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-03-29T10:43:05.010 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-03-29T10:43:05.025 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-03-29T10:43:05.026 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:43:05.039 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-03-29T10:43:05.042 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:43:05.055 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:43:05.071 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:43:05.093 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:43:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:43:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-29T10:43:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:43:05.128 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:43:05.128 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-29T10:43:05.128 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.128 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.128 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.129 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:43:05.129 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:43:05.137 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:43:05.137 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:43:05.137 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:43:05.137 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:43:05.143 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:43:05.143 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:43:05.143 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:43:05.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:43:05.146 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:43:05.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:43:05.286 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:43:05.286 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:43:05.289 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:43:05.289 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:43:05.397 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:43:05.397 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:43:05.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:43:05.413 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:43:05.413 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:43:05.414 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:43:05.426 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:43:05.426 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:43:05.442 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:43:05.442 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:43:05.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:43:05.509 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:43:05.574 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:43:05.588 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:43:05.641 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-03-29T10:43:05.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-03-29T10:43:05.654 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-03-29T10:43:05.655 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-03-29T10:43:05.891 INFO:teuthology.orchestra.run.smithi064.stdout:74 files removed 2024-03-29T10:43:05.903 INFO:teuthology.orchestra.run.smithi174.stdout:74 files removed 2024-03-29T10:43:05.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-03-29T10:43:05.933 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw 2024-03-29T10:43:06.603 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 216 kB/s | 82 kB 00:00 2024-03-29T10:43:06.689 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 180 kB/s | 82 kB 00:00 2024-03-29T10:43:06.928 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-29T10:43:07.131 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T10:43:07.304 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-29T10:43:07.457 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-29T10:43:07.956 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T10:43:08.067 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:43:09.392 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T10:43:09.468 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T10:43:10.450 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T10:43:10.489 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T10:43:12.149 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-03-29T10:43:12.218 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-29T10:43:16.166 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-29T10:43:16.244 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:43:17.542 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 16 kB 00:00 2024-03-29T10:43:17.578 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 16 kB 00:00 2024-03-29T10:43:18.639 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T10:43:18.929 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:43:23.529 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-29T10:43:24.068 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-29T10:43:26.131 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:43:26.143 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:43:26.143 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:43:26.143 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:43:26.143 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:43:26.143 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:43:26.144 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:43:26.145 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:43:26.146 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:43:26.147 INFO:teuthology.orchestra.run.smithi174.stdout:Install 30 Packages 2024-03-29T10:43:26.147 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2024-03-29T10:43:26.147 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:26.147 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 58 M 2024-03-29T10:43:26.147 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:43:26.811 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:43:26.829 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:43:26.829 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:43:26.829 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:43:26.829 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:43:26.830 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:43:26.831 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:43:26.832 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:43:26.833 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-03-29T10:43:26.833 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-03-29T10:43:26.833 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:26.833 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 58 M 2024-03-29T10:43:26.833 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:43:27.738 INFO:teuthology.orchestra.run.smithi174.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.3 MB/s | 5.3 MB 00:01 2024-03-29T10:43:27.813 INFO:teuthology.orchestra.run.smithi174.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 335 kB/s | 25 kB 00:00 2024-03-29T10:43:28.013 INFO:teuthology.orchestra.run.smithi174.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.6 MB/s | 742 kB 00:00 2024-03-29T10:43:28.113 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.9 MB/s | 5.3 MB 00:01 2024-03-29T10:43:28.122 INFO:teuthology.orchestra.run.smithi174.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.5 MB/s | 498 kB 00:00 2024-03-29T10:43:28.146 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 763 kB/s | 25 kB 00:00 2024-03-29T10:43:28.280 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T10:43:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.2 MB/s | 498 kB 00:00 2024-03-29T10:43:29.066 INFO:teuthology.orchestra.run.smithi174.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 5.4 MB/s | 5.1 MB 00:00 2024-03-29T10:43:29.139 INFO:teuthology.orchestra.run.smithi174.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 602 kB/s | 45 kB 00:00 2024-03-29T10:43:29.240 INFO:teuthology.orchestra.run.smithi174.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.3 MB/s | 131 kB 00:00 2024-03-29T10:43:29.323 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 5.7 MB/s | 5.1 MB 00:00 2024-03-29T10:43:29.356 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.3 MB/s | 45 kB 00:00 2024-03-29T10:43:29.406 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T10:43:29.440 INFO:teuthology.orchestra.run.smithi174.stdout:(8/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 6.8 MB/s | 20 MB 00:02 2024-03-29T10:43:29.465 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T10:43:29.465 INFO:teuthology.orchestra.run.smithi174.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 719 kB/s | 162 kB 00:00 2024-03-29T10:43:29.524 INFO:teuthology.orchestra.run.smithi174.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 3.8 MB/s | 320 kB 00:00 2024-03-29T10:43:29.532 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 320 kB 00:00 2024-03-29T10:43:29.549 INFO:teuthology.orchestra.run.smithi174.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T10:43:29.574 INFO:teuthology.orchestra.run.smithi174.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:43:29.599 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-29T10:43:29.657 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T10:43:29.749 INFO:teuthology.orchestra.run.smithi174.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2024-03-29T10:43:29.783 INFO:teuthology.orchestra.run.smithi174.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 863 kB/s | 159 kB 00:00 2024-03-29T10:43:29.841 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 577 kB/s | 106 kB 00:00 2024-03-29T10:43:29.892 INFO:teuthology.orchestra.run.smithi174.stdout:(15/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.9 MB/s | 9.7 MB 00:03 2024-03-29T10:43:29.908 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-29T10:43:29.917 INFO:teuthology.orchestra.run.smithi174.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 264 kB/s | 44 kB 00:00 2024-03-29T10:43:29.933 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-29T10:43:29.975 INFO:teuthology.orchestra.run.smithi174.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 661 kB 00:00 2024-03-29T10:43:30.000 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T10:43:30.000 INFO:teuthology.orchestra.run.smithi174.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 282 kB 00:00 2024-03-29T10:43:30.026 INFO:teuthology.orchestra.run.smithi174.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 379 kB/s | 41 kB 00:00 2024-03-29T10:43:30.042 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-29T10:43:30.051 INFO:teuthology.orchestra.run.smithi174.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T10:43:30.067 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-29T10:43:30.076 INFO:teuthology.orchestra.run.smithi174.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-29T10:43:30.101 INFO:teuthology.orchestra.run.smithi174.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 619 kB/s | 46 kB 00:00 2024-03-29T10:43:30.109 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-03-29T10:43:30.134 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T10:43:30.168 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T10:43:30.352 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T10:43:30.385 INFO:teuthology.orchestra.run.smithi174.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 92 kB/s | 26 kB 00:00 2024-03-29T10:43:30.493 INFO:teuthology.orchestra.run.smithi174.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2024-03-29T10:43:30.518 INFO:teuthology.orchestra.run.smithi174.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-03-29T10:43:30.569 INFO:teuthology.orchestra.run.smithi174.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 899 kB/s | 67 kB 00:00 2024-03-29T10:43:30.694 INFO:teuthology.orchestra.run.smithi174.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-29T10:43:30.761 INFO:teuthology.orchestra.run.smithi174.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.7 MB/s | 4.4 MB 00:00 2024-03-29T10:43:30.786 INFO:teuthology.orchestra.run.smithi174.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 887 kB/s | 192 kB 00:00 2024-03-29T10:43:30.845 INFO:teuthology.orchestra.run.smithi174.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-29T10:43:31.202 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-29T10:43:31.244 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-29T10:43:31.286 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-29T10:43:31.328 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T10:43:31.520 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 20 MB 00:04 2024-03-29T10:43:31.695 INFO:teuthology.orchestra.run.smithi174.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.7 MB/s | 3.4 MB 00:00 2024-03-29T10:43:31.720 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.1 MB/s | 9.7 MB 00:04 2024-03-29T10:43:31.745 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-29T10:43:31.753 INFO:teuthology.orchestra.run.smithi174.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 3.1 MB 00:00 2024-03-29T10:43:31.759 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:31.760 INFO:teuthology.orchestra.run.smithi174.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-29T10:43:31.760 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:43:31.770 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 766 kB/s | 192 kB 00:00 2024-03-29T10:43:31.793 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:43:31.793 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:43:31.954 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-03-29T10:43:32.204 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.2 MB/s | 3.1 MB 00:00 2024-03-29T10:43:32.258 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:43:32.258 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:43:32.338 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.9 MB/s | 3.4 MB 00:00 2024-03-29T10:43:32.342 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:32.342 INFO:teuthology.orchestra.run.smithi064.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-29T10:43:32.342 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:43:32.385 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:43:32.385 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:43:32.875 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:43:32.875 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:43:33.696 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:43:33.794 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:43:33.825 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:43:33.850 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:43:33.873 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:43:33.890 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:43:34.082 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:43:34.136 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:43:34.459 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:43:34.518 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:34.559 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:43:34.593 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:43:34.676 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:43:34.683 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:43:34.701 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:43:34.720 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:43:34.724 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:43:34.740 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:43:34.783 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:43:34.812 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:43:34.855 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:43:34.924 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:43:34.932 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:43:34.947 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:43:34.987 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:43:35.011 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:43:35.046 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:43:35.157 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:43:35.196 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:43:35.237 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:43:35.269 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:43:35.298 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:43:35.322 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:43:35.331 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:43:35.378 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:43:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:43:35.654 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:43:35.659 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:43:35.686 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:43:35.687 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:43:35.732 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:43:35.775 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:43:35.793 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:43:35.807 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:43:35.815 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:43:35.850 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:43:35.862 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:43:35.896 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:43:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:43:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:43:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:43:36.129 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:43:36.158 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:43:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:43:36.302 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:43:36.330 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:43:36.401 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:43:36.469 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:43:36.499 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:43:36.522 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:43:36.554 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:43:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:43:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:43:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:43:37.202 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:43:37.223 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:43:37.294 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:43:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:43:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:43:38.728 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:38.762 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:39.317 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:39.348 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:43:39.441 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:43:39.454 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:43:39.454 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:43:39.454 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:39.474 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:39.660 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:40.274 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:40.308 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:43:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:43:40.383 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:43:40.383 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:43:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:49.188 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:49.188 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-03-29T10:43:49.188 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-03-29T10:43:49.188 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-03-29T10:43:49.188 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-03-29T10:43:49.188 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-03-29T10:43:49.189 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-03-29T10:43:49.189 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-03-29T10:43:49.189 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:50.112 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:43:50.137 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:43:50.137 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:43:50.137 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:43:50.137 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:43:50.138 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:43:50.138 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:50.139 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:43:50.233 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:43:50.233 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:50.443 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:50.443 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-03-29T10:43:50.443 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-03-29T10:43:50.443 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-03-29T10:43:50.444 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-03-29T10:43:50.444 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-03-29T10:43:50.444 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-03-29T10:43:50.444 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-03-29T10:43:50.444 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:43:51.335 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:43:51.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:43:51.337 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:43:51.338 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:51.338 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:43:51.338 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:43:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:43:51.401 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:43:51.401 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:43:51.402 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:43:51.402 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:43:51.402 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:43:51.402 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:51.403 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:43:51.471 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:43:51.471 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:43:52.539 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:43:52.540 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:43:52.541 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:43:52.542 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:43:52.542 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:43:52.542 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:52.542 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:43:52.633 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:52.633 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:43:52.633 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:43:52.633 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:43:52.634 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:43:52.635 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:43:52.636 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:43:52.636 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:43:52.636 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:43:52.636 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:43:52.636 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:43:52.637 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:52.637 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:43:52.637 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:43:52.848 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-test 2024-03-29T10:43:53.405 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:43:53.557 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:43:53.558 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 48 M 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 204 M 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:43:53.716 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:43:53.717 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:43:53.718 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:43:53.719 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:43:53.719 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:43:53.719 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:43:53.719 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:53.719 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:43:53.794 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:43:53.852 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T10:43:53.902 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T10:43:53.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-03-29T10:43:54.053 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 163 kB 00:00 2024-03-29T10:43:54.220 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 590 kB/s | 187 kB 00:00 2024-03-29T10:43:54.560 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:43:54.716 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:43:54.717 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:43:54.718 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 48 M 2024-03-29T10:43:54.719 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 204 M 2024-03-29T10:43:54.719 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:43:55.185 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:43:55.235 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:43:55.269 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:43:55.511 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 775 kB/s | 187 kB 00:00 2024-03-29T10:43:55.536 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 318 kB/s | 163 kB 00:00 2024-03-29T10:43:56.645 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 16 MB/s | 47 MB 00:03 2024-03-29T10:43:56.648 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:56.649 INFO:teuthology.orchestra.run.smithi174.stdout:Total 16 MB/s | 48 MB 00:03 2024-03-29T10:43:56.664 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:43:56.685 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:43:56.685 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:43:57.066 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:43:57.066 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:43:57.643 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:43:57.729 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:43:57.766 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:43:57.798 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:43:57.836 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:43:57.856 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:44:02.304 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:44:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:44:03.977 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 5.3 MB/s | 47 MB 00:08 2024-03-29T10:44:03.980 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:03.981 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.2 MB/s | 48 MB 00:09 2024-03-29T10:44:03.995 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:04.017 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:04.017 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:04.416 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:04.416 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:04.738 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:44:04.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:44:04.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:44:04.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:44:04.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:44:04.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:44:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:44:05.106 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:44:05.137 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:44:05.137 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:05.137 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:05.137 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.137 INFO:teuthology.orchestra.run.smithi174.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:44:05.138 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.138 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:44:05.138 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:44:05.138 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:44:05.138 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:05.138 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:05.146 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:44:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:44:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:44:05.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph 2024-03-29T10:44:06.029 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:06.184 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout:========================================================================================= 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout:========================================================================================= 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:44:06.187 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:44:06.188 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:44:06.189 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout:========================================================================================= 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout:Install 36 Packages 2024-03-29T10:44:06.190 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:06.191 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 30 M 2024-03-29T10:44:06.191 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 107 M 2024-03-29T10:44:06.191 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:06.913 INFO:teuthology.orchestra.run.smithi174.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T10:44:07.347 INFO:teuthology.orchestra.run.smithi174.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2024-03-29T10:44:07.397 INFO:teuthology.orchestra.run.smithi174.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 2.1 MB 00:00 2024-03-29T10:44:07.472 INFO:teuthology.orchestra.run.smithi174.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T10:44:07.598 INFO:teuthology.orchestra.run.smithi174.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:44:07.639 INFO:teuthology.orchestra.run.smithi174.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:44:07.706 INFO:teuthology.orchestra.run.smithi174.stdout:(7/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 5.9 MB/s | 4.7 MB 00:00 2024-03-29T10:44:07.723 INFO:teuthology.orchestra.run.smithi174.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 377 kB/s | 32 kB 00:00 2024-03-29T10:44:07.740 INFO:teuthology.orchestra.run.smithi174.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:44:07.757 INFO:teuthology.orchestra.run.smithi174.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:44:07.783 INFO:teuthology.orchestra.run.smithi174.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:44:07.841 INFO:teuthology.orchestra.run.smithi174.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T10:44:07.866 INFO:teuthology.orchestra.run.smithi174.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T10:44:07.908 INFO:teuthology.orchestra.run.smithi174.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-03-29T10:44:07.934 INFO:teuthology.orchestra.run.smithi174.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-29T10:44:08.076 INFO:teuthology.orchestra.run.smithi174.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 749 kB/s | 125 kB 00:00 2024-03-29T10:44:08.134 INFO:teuthology.orchestra.run.smithi174.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-29T10:44:08.293 INFO:teuthology.orchestra.run.smithi174.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-03-29T10:44:08.318 INFO:teuthology.orchestra.run.smithi174.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 627 kB/s | 151 kB 00:00 2024-03-29T10:44:08.385 INFO:teuthology.orchestra.run.smithi174.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T10:44:08.444 INFO:teuthology.orchestra.run.smithi174.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-29T10:44:08.511 INFO:teuthology.orchestra.run.smithi174.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-29T10:44:08.686 INFO:teuthology.orchestra.run.smithi174.stdout:(23/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-29T10:44:08.702 INFO:teuthology.orchestra.run.smithi174.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 121 kB/s | 23 kB 00:00 2024-03-29T10:44:08.728 INFO:teuthology.orchestra.run.smithi174.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 875 kB/s | 358 kB 00:00 2024-03-29T10:44:08.770 INFO:teuthology.orchestra.run.smithi174.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-29T10:44:08.795 INFO:teuthology.orchestra.run.smithi174.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-03-29T10:44:08.820 INFO:teuthology.orchestra.run.smithi174.stdout:(28/36): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-03-29T10:44:08.845 INFO:teuthology.orchestra.run.smithi174.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T10:44:08.879 INFO:teuthology.orchestra.run.smithi174.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T10:44:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-29T10:44:08.930 INFO:teuthology.orchestra.run.smithi174.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-29T10:44:08.955 INFO:teuthology.orchestra.run.smithi174.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-29T10:44:09.013 INFO:teuthology.orchestra.run.smithi174.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T10:44:09.039 INFO:teuthology.orchestra.run.smithi174.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-29T10:44:09.072 INFO:teuthology.orchestra.run.smithi174.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-29T10:44:09.075 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:09.076 INFO:teuthology.orchestra.run.smithi174.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-29T10:44:09.097 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:09.137 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:09.137 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:09.558 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:09.558 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:44:09.999 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:44:10.909 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:10.967 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:44:11.011 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:44:11.044 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:44:11.114 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:44:11.168 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:44:11.200 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:44:11.292 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:44:11.363 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:44:11.411 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:44:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:44:11.545 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:44:11.583 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:44:11.631 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:44:11.687 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:44:11.871 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:44:11.920 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:44:11.951 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:44:11.986 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:44:12.016 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:44:12.093 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:44:12.134 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:44:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:44:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:44:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:44:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:44:12.374 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:44:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:44:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:44:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:44:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:44:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:44:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:44:12.416 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:44:12.489 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:44:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:44:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:44:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:44:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:12.844 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:12.845 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:12.908 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:44:12.938 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:44:12.963 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:44:12.963 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:12.963 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:44:12.963 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:44:12.963 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:44:12.963 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:13.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-03-29T10:44:13.718 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:49 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:13.879 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:13.881 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-29T10:44:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:13.881 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:44:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:44:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:44:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:13.885 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-03-29T10:44:13.886 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-03-29T10:44:13.886 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:14.388 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:44:14.415 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:44:14.415 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:14.415 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:44:14.415 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:44:14.415 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:44:14.415 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:14.431 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 35 kB/s | 6.6 kB 00:00 2024-03-29T10:44:14.639 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-03-29T10:44:14.707 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.6 MB/s | 2.1 MB 00:00 2024-03-29T10:44:14.765 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.2 MB/s | 247 kB 00:00 2024-03-29T10:44:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:44:14.873 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T10:44:14.879 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:44:14.879 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:14.879 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:44:14.879 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:44:14.879 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:44:14.879 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:14.915 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:44:14.941 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:44:14.974 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:44:15.041 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.7 MB/s | 4.7 MB 00:00 2024-03-29T10:44:15.058 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 462 kB/s | 39 kB 00:00 2024-03-29T10:44:15.084 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:44:15.089 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:44:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:44:15.113 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:15.113 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:44:15.113 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:44:15.113 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:44:15.113 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:15.125 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:44:15.142 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-29T10:44:15.160 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:44:15.178 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:44:15.184 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T10:44:15.260 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-29T10:44:15.352 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 651 kB/s | 125 kB 00:00 2024-03-29T10:44:15.727 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T10:44:15.961 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 216 kB/s | 151 kB 00:00 2024-03-29T10:44:15.986 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 71 kB/s | 43 kB 00:00 2024-03-29T10:44:16.111 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-29T10:44:16.236 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 142 kB/s | 18 kB 00:00 2024-03-29T10:44:16.262 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-29T10:44:16.349 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 277 kB/s | 172 kB 00:00 2024-03-29T10:44:16.370 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 173 kB/s | 23 kB 00:00 2024-03-29T10:44:16.387 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-29T10:44:16.463 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 173 kB/s | 20 kB 00:00 2024-03-29T10:44:16.496 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-03-29T10:44:16.605 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-29T10:44:16.630 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 325 kB/s | 79 kB 00:00 2024-03-29T10:44:16.672 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T10:44:16.724 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 754 kB/s | 90 kB 00:00 2024-03-29T10:44:16.758 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 280 kB/s | 36 kB 00:00 2024-03-29T10:44:16.808 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-29T10:44:16.867 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-29T10:44:16.892 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-29T10:44:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:16.897 INFO:teuthology.orchestra.run.smithi064.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-03-29T10:44:16.904 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:44:16.904 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:44:16.904 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:44:16.904 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:44:16.904 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:44:16.904 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:44:16.905 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:44:16.906 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:44:16.907 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:44:16.907 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:44:16.907 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:44:16.907 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:44:16.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:44:16.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:44:16.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:44:16.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:44:16.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:44:16.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:44:16.923 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:16.969 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:16.969 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:17.397 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:17.397 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:17.908 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:44:17.909 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:44:17.910 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:44:17.911 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:44:17.911 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:44:17.911 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:17.911 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:18.248 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-base 2024-03-29T10:44:18.746 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:18.803 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:44:18.836 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:18.839 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:44:18.864 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:44:18.935 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:18.936 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:44:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:44:18.986 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:18.988 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:44:18.988 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:44:19.068 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install cephadm 2024-03-29T10:44:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:44:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:44:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:44:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:44:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:44:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:44:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:44:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:44:19.628 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:44:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:44:19.777 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 747 k 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 754 k 2024-03-29T10:44:19.778 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:44:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:44:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:44:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:44:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:44:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:44:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:44:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:44:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:44:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:44:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:44:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:44:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:44:20.572 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:44:20.683 INFO:teuthology.orchestra.run.smithi174.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 833 kB/s | 747 kB 00:00 2024-03-29T10:44:20.683 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:20.683 INFO:teuthology.orchestra.run.smithi174.stdout:Total 831 kB/s | 747 kB 00:00 2024-03-29T10:44:20.683 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:20.684 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:20.684 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:20.690 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:20.690 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:44:20.765 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:44:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:44:20.801 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:20.802 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:44:20.802 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:44:20.802 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:44:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:21.234 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:21.330 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:21.700 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:21.985 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:21.986 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:21.986 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:21.986 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:21.986 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:21.986 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:22.230 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:44:22.326 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:44:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:44:22.353 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:22.353 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:44:22.353 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:44:22.353 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:44:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:22.806 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:44:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:44:22.842 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:22.842 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:44:22.842 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:44:22.842 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:44:22.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:22.960 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:22.961 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-03-29T10:44:22.961 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:22.961 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:44:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 145 k 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 436 k 2024-03-29T10:44:22.962 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:44:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:44:23.084 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:23.084 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:44:23.084 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:44:23.084 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:44:23.084 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:23.153 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:44:23.356 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 369 kB/s | 145 kB 00:00 2024-03-29T10:44:23.357 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:23.357 INFO:teuthology.orchestra.run.smithi174.stdout:Total 367 kB/s | 145 kB 00:00 2024-03-29T10:44:23.357 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:23.365 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:23.365 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:23.419 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:23.419 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:23.562 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:23.624 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:23.650 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:23.650 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:23.650 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:44:23.650 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:24.621 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:24.621 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:24.622 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:24.622 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:24.622 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:24.622 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:24.811 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:44:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:44:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:44:24.814 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:44:24.815 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:44:24.816 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:44:24.913 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr 2024-03-29T10:44:25.491 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:25.591 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:25.644 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:25.645 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:44:25.645 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:44:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:25.698 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:44:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:25.701 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:25.725 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:44:26.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-03-29T10:44:26.286 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:26.437 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:44:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout:Install 12 Packages 2024-03-29T10:44:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:26.440 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 6.9 M 2024-03-29T10:44:26.440 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 26 M 2024-03-29T10:44:26.440 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:26.623 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:26.780 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:26.781 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:44:26.781 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:26.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-03-29T10:44:27.159 INFO:teuthology.orchestra.run.smithi174.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 121 kB/s | 31 kB 00:00 2024-03-29T10:44:27.185 INFO:teuthology.orchestra.run.smithi174.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 54 kB/s | 15 kB 00:00 2024-03-29T10:44:27.327 INFO:teuthology.orchestra.run.smithi174.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T10:44:27.377 INFO:teuthology.orchestra.run.smithi174.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T10:44:27.425 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:27.494 INFO:teuthology.orchestra.run.smithi174.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.0 MB/s | 1.8 MB 00:00 2024-03-29T10:44:27.582 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:27.583 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:27.583 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:27.583 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:27.583 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:27.583 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:44:27.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 747 k 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 754 k 2024-03-29T10:44:27.584 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:27.644 INFO:teuthology.orchestra.run.smithi174.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-29T10:44:27.811 INFO:teuthology.orchestra.run.smithi174.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 61 kB/s | 19 kB 00:00 2024-03-29T10:44:27.861 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T10:44:27.862 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:27.862 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T10:44:27.862 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:27.867 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:27.868 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:27.873 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:27.874 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:27.894 INFO:teuthology.orchestra.run.smithi174.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T10:44:27.947 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:27.961 INFO:teuthology.orchestra.run.smithi174.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 462 kB/s | 31 kB 00:00 2024-03-29T10:44:28.062 INFO:teuthology.orchestra.run.smithi174.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T10:44:28.103 INFO:teuthology.orchestra.run.smithi174.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.4 MB/s | 2.0 MB 00:00 2024-03-29T10:44:28.129 INFO:teuthology.orchestra.run.smithi174.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 455 kB/s | 144 kB 00:00 2024-03-29T10:44:28.133 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:28.134 INFO:teuthology.orchestra.run.smithi174.stdout:Total 4.0 MB/s | 6.9 MB 00:01 2024-03-29T10:44:28.152 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:28.168 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:28.169 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:28.344 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:28.344 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:28.470 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:28.946 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:44:28.969 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:28.981 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:44:29.036 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:44:29.083 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:44:29.186 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:44:29.261 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:29.261 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:29.261 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:29.261 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:29.261 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:29.261 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:29.308 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:44:29.402 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:44:29.446 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:44:29.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:44:29.474 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:44:29.514 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:44:29.755 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:44:29.797 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:44:30.032 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:30.190 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:30.191 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:30.191 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T10:44:30.191 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:30.191 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 436 k 2024-03-29T10:44:30.192 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:30.712 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:44:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:44:30.759 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 256 kB/s | 145 kB 00:00 2024-03-29T10:44:30.760 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:30.760 INFO:teuthology.orchestra.run.smithi064.stdout:Total 256 kB/s | 145 kB 00:00 2024-03-29T10:44:30.760 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:30.768 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:30.768 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:30.822 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:30.823 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:31.069 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:44:31.069 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:31.069 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:31.069 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:44:31.069 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:31.070 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:31.088 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:31.167 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:31.168 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:31.168 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:44:31.168 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:31.331 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:44:31.896 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:32.048 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:32.049 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-29T10:44:32.049 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:32.049 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-29T10:44:32.049 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:44:32.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:44:32.051 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:32.051 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:32.051 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2024-03-29T10:44:32.051 INFO:teuthology.orchestra.run.smithi174.stdout:Install 12 Packages 2024-03-29T10:44:32.051 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:32.051 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 37 M 2024-03-29T10:44:32.052 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 209 M 2024-03-29T10:44:32.052 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:32.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:32.109 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:32.109 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:32.109 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:32.109 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:32.109 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:32.376 INFO:teuthology.orchestra.run.smithi174.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 32 kB 00:00 2024-03-29T10:44:32.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-03-29T10:44:32.409 INFO:teuthology.orchestra.run.smithi174.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-03-29T10:44:32.460 INFO:teuthology.orchestra.run.smithi174.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 698 kB/s | 34 kB 00:00 2024-03-29T10:44:32.518 INFO:teuthology.orchestra.run.smithi174.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T10:44:32.710 INFO:teuthology.orchestra.run.smithi174.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T10:44:32.777 INFO:teuthology.orchestra.run.smithi174.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:44:32.802 INFO:teuthology.orchestra.run.smithi174.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:44:32.969 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:33.072 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:33.128 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:33.129 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:44:33.129 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:33.136 INFO:teuthology.orchestra.run.smithi174.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T10:44:33.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:44:33.353 INFO:teuthology.orchestra.run.smithi174.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-29T10:44:33.479 INFO:teuthology.orchestra.run.smithi174.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 571 kB/s | 195 kB 00:00 2024-03-29T10:44:33.570 INFO:teuthology.orchestra.run.smithi174.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.5 MB/s | 7.4 MB 00:01 2024-03-29T10:44:33.670 INFO:teuthology.orchestra.run.smithi174.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 776 kB/s | 245 kB 00:00 2024-03-29T10:44:33.674 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:33.674 INFO:teuthology.orchestra.run.smithi174.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-29T10:44:33.713 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:33.736 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:33.737 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:33.776 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:33.933 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:33.934 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-29T10:44:33.934 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:44:33.935 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:44:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:44:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:33.936 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:33.936 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-29T10:44:33.936 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-29T10:44:33.936 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:33.937 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.9 M 2024-03-29T10:44:33.937 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-29T10:44:33.938 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:34.097 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:34.097 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:34.413 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 61 kB/s | 15 kB 00:00 2024-03-29T10:44:34.438 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 114 kB/s | 31 kB 00:00 2024-03-29T10:44:34.563 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.4 MB/s | 1.8 MB 00:00 2024-03-29T10:44:34.622 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T10:44:34.639 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:34.859 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:44:34.889 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-03-29T10:44:34.891 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:44:34.906 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-29T10:44:35.017 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:44:35.056 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T10:44:35.061 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:44:35.081 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 824 kB/s | 144 kB 00:00 2024-03-29T10:44:35.124 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 459 kB/s | 31 kB 00:00 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T10:44:35.237 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:44:35.268 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:44:35.350 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:44:35.516 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-29T10:44:35.521 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:35.521 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.3 MB/s | 6.9 MB 00:01 2024-03-29T10:44:35.540 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:35.557 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:35.557 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:35.733 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:35.733 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:44:36.114 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:44:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:36.325 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:44:36.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:44:36.409 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:44:36.538 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:44:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:44:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:44:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:44:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:44:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:44:37.141 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:44:37.176 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:44:37.355 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:44:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:44:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:44:38.336 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:44:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:44:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:44:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:44:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:44:38.337 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:38.337 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:38.380 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:44:38.408 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:44:38.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:44:39.123 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:39.283 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:39.284 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-29T10:44:39.284 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:39.284 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-29T10:44:39.284 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:39.284 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:39.285 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:39.286 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-29T10:44:39.286 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-29T10:44:39.286 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:39.286 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 37 M 2024-03-29T10:44:39.286 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-03-29T10:44:39.286 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:39.589 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 215 kB/s | 32 kB 00:00 2024-03-29T10:44:39.622 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2024-03-29T10:44:39.665 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2024-03-29T10:44:39.715 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:44:39.907 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:44:39.973 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:44:39.999 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:44:40.207 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-29T10:44:40.366 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-29T10:44:40.391 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T10:44:40.917 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 446 kB/s | 245 kB 00:00 2024-03-29T10:44:40.956 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:44:40.956 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:44:40.956 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:44:40.956 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:44:40.956 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:44:40.957 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:44:41.042 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.6 MB/s | 7.4 MB 00:01 2024-03-29T10:44:41.044 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:41.044 INFO:teuthology.orchestra.run.smithi064.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T10:44:41.085 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:41.111 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:41.111 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:41.403 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:44:41.403 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:41.403 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:41.403 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:41.404 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:41.490 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:41.490 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:41.802 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:44:42.206 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:42.274 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:44:42.305 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:44:42.376 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:42.407 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:44:42.450 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:44:42.528 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================== 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================== 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:44:42.529 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================== 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout:Install 13 Packages 2024-03-29T10:44:42.530 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:42.531 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 2.0 M 2024-03-29T10:44:42.531 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 26 M 2024-03-29T10:44:42.531 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:42.634 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:44:42.665 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:44:42.747 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:44:42.878 INFO:teuthology.orchestra.run.smithi174.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-29T10:44:42.908 INFO:teuthology.orchestra.run.smithi174.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 95 kB/s | 18 kB 00:00 2024-03-29T10:44:42.967 INFO:teuthology.orchestra.run.smithi174.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-29T10:44:43.000 INFO:teuthology.orchestra.run.smithi174.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T10:44:43.025 INFO:teuthology.orchestra.run.smithi174.stdout:(5/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 170 kB/s | 52 kB 00:00 2024-03-29T10:44:43.067 INFO:teuthology.orchestra.run.smithi174.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 211 kB 00:00 2024-03-29T10:44:43.092 INFO:teuthology.orchestra.run.smithi174.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 464 kB/s | 43 kB 00:00 2024-03-29T10:44:43.438 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:44:43.564 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:44:43.626 INFO:teuthology.orchestra.run.smithi174.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 27 kB/s | 15 kB 00:00 2024-03-29T10:44:43.668 INFO:teuthology.orchestra.run.smithi174.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 50 kB/s | 32 kB 00:00 2024-03-29T10:44:43.835 INFO:teuthology.orchestra.run.smithi174.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 273 kB/s | 203 kB 00:00 2024-03-29T10:44:43.902 INFO:teuthology.orchestra.run.smithi174.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-03-29T10:44:43.969 INFO:teuthology.orchestra.run.smithi174.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 672 kB/s | 90 kB 00:00 2024-03-29T10:44:44.352 INFO:teuthology.orchestra.run.smithi174.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-03-29T10:44:44.354 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:44.355 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-03-29T10:44:44.355 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:44.370 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:44.370 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:44.468 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:44.469 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:44.872 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:44:44.992 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:45.181 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:44:45.236 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:44:45.291 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:44:45.329 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:44:45.370 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:44:45.463 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:44:45.563 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:44:45.621 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:44:45.941 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:44:45.964 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:44:46.342 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:44:46.385 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:44:46.418 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:44:46.467 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:44:46.488 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:44:47.252 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:44:47.253 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:44:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:44:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:47.617 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:47.618 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:47.926 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:44:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:44:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:44:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:44:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:44:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:44:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:44:48.494 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:48.646 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:48.646 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================== 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================== 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:44:48.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================== 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 6.8 M 2024-03-29T10:44:48.648 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 33 M 2024-03-29T10:44:48.649 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:44:48.836 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:48.837 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:49.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:44:49.371 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 605 kB/s | 138 kB 00:00 2024-03-29T10:44:49.421 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 891 kB/s | 248 kB 00:00 2024-03-29T10:44:49.646 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-29T10:44:49.696 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 210 kB/s | 58 kB 00:00 2024-03-29T10:44:49.768 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:49.896 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T10:44:49.922 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 185 kB/s | 51 kB 00:00 2024-03-29T10:44:49.922 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:49.923 INFO:teuthology.orchestra.run.smithi174.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-29T10:44:49.927 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:49.928 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:44:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-03-29T10:44:49.930 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:49.931 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.0 M 2024-03-29T10:44:49.931 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-29T10:44:49.931 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:49.936 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:49.947 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:49.947 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:50.035 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:50.035 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:50.149 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 24 kB 00:00 2024-03-29T10:44:50.166 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-03-29T10:44:50.216 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T10:44:50.241 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:44:50.266 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:44:50.283 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:44:50.309 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 185 kB/s | 52 kB 00:00 2024-03-29T10:44:50.350 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:50.504 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:44:50.518 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-29T10:44:50.564 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:44:50.568 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-29T10:44:50.776 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-03-29T10:44:50.827 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 392 kB/s | 203 kB 00:00 2024-03-29T10:44:50.952 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-29T10:44:51.060 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T10:44:51.062 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:51.063 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-29T10:44:51.063 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:51.079 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:51.079 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:51.085 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:44:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:44:51.181 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:51.181 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:51.237 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:44:51.260 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:44:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:44:51.940 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:44:51.998 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:44:52.030 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:44:52.066 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:44:52.067 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:44:52.067 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:44:52.067 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:44:52.067 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:44:52.067 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:44:52.067 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:44:52.162 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:44:52.273 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:44:52.324 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:44:52.383 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:52.384 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:52.648 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-fuse 2024-03-29T10:44:53.090 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:44:53.136 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:44:53.170 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:44:53.213 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:53.219 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:44:53.248 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:44:53.368 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:53.369 INFO:teuthology.orchestra.run.smithi174.stdout:Install 2 Packages 2024-03-29T10:44:53.370 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:53.370 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 964 k 2024-03-29T10:44:53.370 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 2.8 M 2024-03-29T10:44:53.370 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:53.639 INFO:teuthology.orchestra.run.smithi174.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 292 kB/s | 79 kB 00:00 2024-03-29T10:44:54.066 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:44:54.067 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:44:54.114 INFO:teuthology.orchestra.run.smithi174.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T10:44:54.115 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:54.116 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T10:44:54.116 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:54.136 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:54.136 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:54.209 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:54.210 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:54.390 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:44:54.435 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:54.436 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:54.523 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:44:54.597 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:44:54.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:44:55.246 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:55.260 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:44:55.261 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:44:55.405 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:44:55.406 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-29T10:44:55.407 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:55.408 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.8 M 2024-03-29T10:44:55.409 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-03-29T10:44:55.409 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:55.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:44:55.591 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:55.591 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:55.591 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:44:55.591 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:55.591 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:55.838 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T10:44:55.854 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-volume 2024-03-29T10:44:55.980 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-29T10:44:56.080 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 334 kB/s | 138 kB 00:00 2024-03-29T10:44:56.147 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-29T10:44:56.164 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 316 kB/s | 58 kB 00:00 2024-03-29T10:44:56.181 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2024-03-29T10:44:56.183 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:56.184 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.7 MB/s | 6.8 MB 00:00 2024-03-29T10:44:56.197 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:44:56.208 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:44:56.208 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:44:56.300 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:44:56.300 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:44:56.430 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:56.586 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:56.587 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:56.588 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:56.588 INFO:teuthology.orchestra.run.smithi174.stdout:Install 2 Packages 2024-03-29T10:44:56.588 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:56.588 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 332 k 2024-03-29T10:44:56.588 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 1.5 M 2024-03-29T10:44:56.588 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:56.616 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:44:56.710 INFO:teuthology.orchestra.run.smithi174.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 567 kB/s | 69 kB 00:00 2024-03-29T10:44:56.788 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:44:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:44:56.863 INFO:teuthology.orchestra.run.smithi174.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 972 kB/s | 264 kB 00:00 2024-03-29T10:44:56.863 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:56.863 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T10:44:56.863 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:56.870 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:56.870 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:56.907 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:56.907 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:57.240 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:44:57.271 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:44:57.286 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:44:57.287 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:57.287 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:44:57.287 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:44:57.480 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:44:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:44:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:44:58.002 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:44:58.338 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:44:58.338 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:58.338 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:44:58.339 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:58.339 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:44:58.339 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:58.339 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:44:58.383 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:44:58.383 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:44:58.383 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:44:58.383 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:44:58.383 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:44:58.383 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:44:58.707 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:58.708 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:44:58.713 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados-devel 2024-03-29T10:44:58.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-03-29T10:44:59.289 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:44:59.439 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:44:59.440 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:59.440 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-03-29T10:44:59.440 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:59.440 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:44:59.440 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 127 k 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 456 k 2024-03-29T10:44:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:44:59.518 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:44:59.674 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:44:59.675 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 964 k 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-03-29T10:44:59.676 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:44:59.690 INFO:teuthology.orchestra.run.smithi174.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 509 kB/s | 127 kB 00:00 2024-03-29T10:44:59.690 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:59.691 INFO:teuthology.orchestra.run.smithi174.stdout:Total 506 kB/s | 127 kB 00:00 2024-03-29T10:44:59.691 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:44:59.696 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:44:59.696 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:44:59.751 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:44:59.752 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:44:59.799 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 642 kB/s | 79 kB 00:00 2024-03-29T10:44:59.896 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:44:59.999 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T10:45:00.000 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:00.000 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.9 MB/s | 964 kB 00:00 2024-03-29T10:45:00.001 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:00.005 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:00.021 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:00.021 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:00.098 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:00.098 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:00.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:45:00.455 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:00.510 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:45:00.782 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:00.782 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:00.782 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:45:00.782 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:00.782 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:00.782 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:01.032 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs2 2024-03-29T10:45:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:45:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:45:01.491 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:45:01.491 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:01.491 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:01.491 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:45:01.491 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:01.491 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:01.609 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:01.711 INFO:teuthology.orchestra.run.smithi174.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:01.765 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:01.766 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:01.766 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:01.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-03-29T10:45:01.849 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs-devel 2024-03-29T10:45:02.357 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:02.421 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:02.516 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:02.517 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:45:02.518 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:02.518 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-29T10:45:02.518 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:02.518 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 332 k 2024-03-29T10:45:02.518 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-03-29T10:45:02.518 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:45:02.574 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:02.574 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 31 k 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 140 k 2024-03-29T10:45:02.575 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:45:02.638 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 577 kB/s | 69 kB 00:00 2024-03-29T10:45:02.751 INFO:teuthology.orchestra.run.smithi174.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 178 kB/s | 31 kB 00:00 2024-03-29T10:45:02.752 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:02.752 INFO:teuthology.orchestra.run.smithi174.stdout:Total 176 kB/s | 31 kB 00:00 2024-03-29T10:45:02.752 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:45:02.755 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:45:02.755 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:45:02.778 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:45:02.778 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:45:02.796 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 950 kB/s | 264 kB 00:00 2024-03-29T10:45:02.797 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:02.798 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T10:45:02.798 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:02.807 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:02.807 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:02.849 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:02.849 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:02.929 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:45:03.004 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:03.032 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:03.176 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:45:03.215 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:45:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:45:03.231 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:03.231 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:45:03.231 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:03.425 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:03.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:03.697 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:03.697 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:45:03.697 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:03.697 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:03.698 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:03.885 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados2 2024-03-29T10:45:04.016 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:45:04.365 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:45:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:04.366 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:45:04.366 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:04.366 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:04.447 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:04.543 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:04.593 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:04.594 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:04.594 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:04.644 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librbd1 2024-03-29T10:45:04.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-03-29T10:45:05.207 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:05.262 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:05.304 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:05.356 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:05.357 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:05.357 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:05.418 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:45:05.419 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:05.420 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:45:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:05.420 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-03-29T10:45:05.420 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-03-29T10:45:05.420 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:45:05.422 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rados 2024-03-29T10:45:05.736 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 402 kB/s | 127 kB 00:00 2024-03-29T10:45:05.736 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:05.737 INFO:teuthology.orchestra.run.smithi064.stdout:Total 400 kB/s | 127 kB 00:00 2024-03-29T10:45:05.737 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:05.743 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:05.743 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:05.801 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:05.801 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:05.958 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:05.988 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:06.063 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:06.086 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:06.138 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:06.139 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:06.139 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:06.185 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rgw 2024-03-29T10:45:06.514 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:06.748 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:06.837 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:06.837 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:06.837 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:06.837 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:06.838 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:06.838 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:06.846 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:06.898 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:06.899 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:06.899 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:06.952 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-cephfs 2024-03-29T10:45:07.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-03-29T10:45:07.514 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:07.611 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:07.662 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:07.663 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:07.664 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:07.664 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:07.709 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rbd 2024-03-29T10:45:07.764 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:07.821 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:07.822 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:07.822 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:07.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-03-29T10:45:08.293 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:08.398 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:08.454 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:08.454 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-03-29T10:45:08.455 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:08.477 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:08.509 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-fuse 2024-03-29T10:45:08.635 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:08.635 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 k 2024-03-29T10:45:08.636 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 140 k 2024-03-29T10:45:08.637 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:45:08.947 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 100 kB/s | 31 kB 00:00 2024-03-29T10:45:08.948 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:08.948 INFO:teuthology.orchestra.run.smithi064.stdout:Total 100 kB/s | 31 kB 00:00 2024-03-29T10:45:08.948 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:08.951 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:08.951 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:08.986 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:08.986 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:09.087 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:09.116 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:09.223 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:09.238 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:09.238 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 85 k 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 231 k 2024-03-29T10:45:09.239 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:45:09.454 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 399 kB/s | 85 kB 00:00 2024-03-29T10:45:09.454 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:09.455 INFO:teuthology.orchestra.run.smithi174.stdout:Total 395 kB/s | 85 kB 00:00 2024-03-29T10:45:09.455 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:45:09.462 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:45:09.462 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:45:09.517 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:45:09.518 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:45:09.634 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:09.654 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:45:09.752 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:09.896 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:09.897 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:09.897 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:09.897 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:10.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-03-29T10:45:10.125 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:10.419 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:10.419 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:10.419 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:45:10.419 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:10.419 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:10.419 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:10.646 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-mirror 2024-03-29T10:45:10.664 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:10.763 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:10.818 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:10.819 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:10.819 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:10.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-03-29T10:45:11.229 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:11.384 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:45:11.385 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:11.386 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.1 M 2024-03-29T10:45:11.386 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 13 M 2024-03-29T10:45:11.386 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:45:11.438 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:11.538 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:11.595 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:11.596 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:11.596 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:11.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-03-29T10:45:11.923 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.8 MB/s | 3.1 MB 00:00 2024-03-29T10:45:11.923 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:11.924 INFO:teuthology.orchestra.run.smithi174.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-29T10:45:11.924 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:45:11.933 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:45:11.933 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:45:12.006 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:45:12.006 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:45:12.214 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:12.315 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:12.371 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:12.505 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:45:12.506 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:12.506 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:12.507 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:12.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-03-29T10:45:12.523 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:12.523 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:12.523 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:45:12.523 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:45:12.524 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:45:12.524 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:13.087 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:13.189 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:13.244 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:13.245 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:13.245 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:13.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-03-29T10:45:13.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:13.629 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:13.629 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:45:13.629 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:13.629 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:13.629 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:13.881 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:13.897 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-nbd 2024-03-29T10:45:13.985 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:14.040 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:14.041 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:14.041 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:14.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-03-29T10:45:14.474 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:43:23 AM UTC. 2024-03-29T10:45:14.626 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:14.627 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 171 k 2024-03-29T10:45:14.628 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 494 k 2024-03-29T10:45:14.628 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-29T10:45:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:14.774 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:14.828 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:14.829 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-29T10:45:14.829 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:14.850 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 769 kB/s | 171 kB 00:00 2024-03-29T10:45:14.851 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:14.852 INFO:teuthology.orchestra.run.smithi174.stdout:Total 764 kB/s | 171 kB 00:00 2024-03-29T10:45:14.852 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-29T10:45:14.859 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-29T10:45:14.859 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-29T10:45:14.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-03-29T10:45:14.914 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-29T10:45:14.915 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-29T10:45:15.062 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-29T10:45:15.158 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:15.461 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:15.605 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:15.615 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:15.616 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:45:15.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:15.617 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-03-29T10:45:15.617 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-03-29T10:45:15.617 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:45:15.904 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:15.904 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:15.905 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-29T10:45:15.905 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:15.905 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:15.905 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-29T10:45:15.935 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 268 kB/s | 85 kB 00:00 2024-03-29T10:45:15.936 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:15.936 INFO:teuthology.orchestra.run.smithi064.stdout:Total 267 kB/s | 85 kB 00:00 2024-03-29T10:45:15.936 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:15.943 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:15.944 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:16.001 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:16.001 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:16.157 DEBUG:teuthology.parallel:result is None 2024-03-29T10:45:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:16.635 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:16.930 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:16.930 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:16.931 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:16.931 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:16.931 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:16.931 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:17.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-03-29T10:45:17.750 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:17.909 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:17.910 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:45:17.911 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:17.911 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:45:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:17.911 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-03-29T10:45:17.911 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-03-29T10:45:17.911 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:45:18.447 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.8 MB/s | 3.1 MB 00:00 2024-03-29T10:45:18.448 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:18.448 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-29T10:45:18.449 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:18.458 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:18.459 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:18.535 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:18.536 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:19.090 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:19.090 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:45:19.090 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:45:19.090 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:45:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:20.320 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:20.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:20.321 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:20.321 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:20.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:20.321 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:20.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-03-29T10:45:21.149 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 10:43:24 AM UTC. 2024-03-29T10:45:21.305 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-29T10:45:21.305 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:21.305 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 171 k 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-03-29T10:45:21.306 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-29T10:45:21.784 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 359 kB/s | 171 kB 00:00 2024-03-29T10:45:21.785 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:21.785 INFO:teuthology.orchestra.run.smithi064.stdout:Total 358 kB/s | 171 kB 00:00 2024-03-29T10:45:21.785 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-29T10:45:21.794 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-29T10:45:21.794 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-29T10:45:21.854 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-29T10:45:21.854 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-29T10:45:22.006 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-29T10:45:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:22.545 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:22.829 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-29T10:45:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:22.829 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:22.829 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-29T10:45:23.063 DEBUG:teuthology.parallel:result is None 2024-03-29T10:45:23.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:45:23.297 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:45:23.317 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:45:23.317 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:45:23.318 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:45:23.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:45:23.478 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:45:23.504 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:45:23.504 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:45:23.504 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:45:23.506 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:45:23.506 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:23.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:45:23.533 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:23.534 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:45:23.576 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:45:23.576 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:23.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:45:23.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:45:23.669 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:23.669 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:45:23.698 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:45:23.768 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:45:23.768 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:23.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:45:23.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:45:23.862 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:23.862 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:45:23.894 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:45:23.964 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:45:23.964 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:23.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:45:23.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:45:24.057 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:24.058 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:45:24.089 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:45:24.159 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:45:24.243 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:45:24.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-03-29T10:45:24.246 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2024-03-29T10:45:24.275 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:45:24.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:45:24.314 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:45:24.344 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:45:24.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:45:24.381 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:45:24.423 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:45:24.423 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:45:24.423 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622655', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1XYCHPDgbbRvcQkzvM0nSiBLU6dxAKguv4WJ+e8cGFf0JzLEoLa3QcVarrIPCZovi3uCVThv/ja51YOi9SH7k=', 'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJI+GBK6tFBCw/9r9EfQ2evtADR04/h6CTSao5wyoWR+iyOenbhJP8D8WKEC+5hpebwKZf8C1TUhmL9pYHsv20c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:45:24.424 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:45:24.448 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:45:24.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:45:24.502 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:45:24.537 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:24.537 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:45:24.581 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:45:24.581 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-03-29T10:45:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 10:45:19.372714922 +0000 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 10:42:42.381998848 +0000 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 10:42:42.381998848 +0000 2024-03-29T10:45:24.636 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 10:42:42.381998848 +0000 2024-03-29T10:45:24.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:45:24.700 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T10:45:24.700 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T10:45:24.701 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000233873 s, 2.2 MB/s 2024-03-29T10:45:24.701 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:45:24.757 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-03-29T10:45:24.811 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 10:45:19.372714922 +0000 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 10:42:42.727992131 +0000 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 10:42:42.727992131 +0000 2024-03-29T10:45:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 10:42:42.727992131 +0000 2024-03-29T10:45:24.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:45:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T10:45:24.876 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T10:45:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000225581 s, 2.3 MB/s 2024-03-29T10:45:24.877 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:45:24.932 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 10:45:19.372714922 +0000 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 10:42:43.064985589 +0000 2024-03-29T10:45:24.988 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 10:42:43.064985589 +0000 2024-03-29T10:45:24.988 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 10:42:43.064985589 +0000 2024-03-29T10:45:24.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:45:25.052 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T10:45:25.053 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T10:45:25.053 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000203222 s, 2.5 MB/s 2024-03-29T10:45:25.054 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:45:25.109 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-29 10:45:19.373714903 +0000 2024-03-29T10:45:25.164 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-29 10:42:43.404978989 +0000 2024-03-29T10:45:25.165 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-29 10:42:43.404978989 +0000 2024-03-29T10:45:25.165 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-29 10:42:43.404978989 +0000 2024-03-29T10:45:25.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:45:25.229 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-29T10:45:25.229 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-29T10:45:25.229 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000199109 s, 2.6 MB/s 2024-03-29T10:45:25.230 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:45:25.286 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:25.286 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:25.286 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:45:25.305 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:45:25.305 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-03-29T10:45:25.362 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:45:25.362 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-29 10:45:12.791558406 +0000 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-29 10:42:40.988949008 +0000 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-29 10:42:40.988949008 +0000 2024-03-29T10:45:25.363 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-29 10:42:40.988949008 +0000 2024-03-29T10:45:25.364 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:45:25.431 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-29T10:45:25.431 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-29T10:45:25.432 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000277352 s, 1.8 MB/s 2024-03-29T10:45:25.433 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:45:25.492 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-03-29T10:45:25.548 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-29 10:45:12.792558392 +0000 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-29 10:42:41.339944227 +0000 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-29 10:42:41.339944227 +0000 2024-03-29T10:45:25.549 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-29 10:42:41.339944227 +0000 2024-03-29T10:45:25.549 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:45:25.616 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-29T10:45:25.617 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-29T10:45:25.617 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000174769 s, 2.9 MB/s 2024-03-29T10:45:25.618 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:45:25.675 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-29 10:45:12.792558392 +0000 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-29 10:42:41.676939638 +0000 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-29 10:42:41.676939638 +0000 2024-03-29T10:45:25.731 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-29 10:42:41.676939638 +0000 2024-03-29T10:45:25.731 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:45:25.798 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-29T10:45:25.798 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-29T10:45:25.798 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000240985 s, 2.1 MB/s 2024-03-29T10:45:25.799 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:45:25.857 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-03-29T10:45:25.911 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-29 10:45:12.793558379 +0000 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-29 10:42:42.016935006 +0000 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-29 10:42:42.016935006 +0000 2024-03-29T10:45:25.912 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-29 10:42:42.016935006 +0000 2024-03-29T10:45:25.912 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:45:25.978 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-29T10:45:25.978 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-29T10:45:25.978 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000222178 s, 2.3 MB/s 2024-03-29T10:45:25.979 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:45:26.035 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:26.035 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:45:26.035 INFO:tasks.ceph:Generating config... 2024-03-29T10:45:26.036 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:45:26.036 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:45:26.036 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:45:26.036 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:45:26.037 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:45:26.037 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:45:26.037 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:45:26.037 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:45:26.037 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:45:26.037 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:45:26.037 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:45:26.038 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:45:26.038 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:45:26.038 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:45:26.038 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:45:26.038 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:45:26.039 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:45:26.040 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:45:26.040 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:45:26.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:45:26.091 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:45:26.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:45:26.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:45:26.166 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.64:6789]'), ('mon.c', '[v1:172.21.15.64:6790]'), ('mon.b', '[v1:172.21.15.174:6789]')] 2024-03-29T10:45:26.166 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.174:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:45:26.167 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:26.167 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:45:26.221 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.64:6789]' --addv c '[v1:172.21.15.64:6790]' --addv b '[v1:172.21.15.174:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2024-03-29T10:45:26.306 INFO:teuthology.orchestra.run.smithi064.stdout:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:45:26.307 INFO:teuthology.orchestra.run.smithi064.stdout:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:45:26.307 INFO:teuthology.orchestra.run.smithi064.stdout:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:45:26.307 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:45:26.308 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:45:26.322 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0263dc76-f748-4029-877b-c4b256090311... 2024-03-29T10:45:26.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:45:26.365 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:45:26.412 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-03-29T10:45:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-03-29T10:45:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-03-29T10:45:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:45:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-03-29T10:45:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.174:6789]" 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: lockdep = True 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 1 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr2 = False 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.416 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-03-29T10:45:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = zstd 2024-03-29T10:45:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: osd class default list = * 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: osd class load list = * 2024-03-29T10:45:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: osd client watch timeout = 120 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-29T10:45:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-29T10:45:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:45:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-03-29T10:45:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: debug objecter = 20 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: debug rados = 20 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-03-29T10:45:26.425 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:45:26.427 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:45:26.428 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.428 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2024-03-29T10:45:26.428 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.428 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2024-03-29T10:45:26.428 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2024-03-29T10:45:26.428 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2024-03-29T10:45:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.174:6789]" 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: lockdep = True 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: mon client directed command retry = 5 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: mon election default strategy = 1 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: ms bind msgr2 = False 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject socket failures = 5000 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: ms type = async 2024-03-29T10:45:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2024-03-29T10:45:26.431 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug trim objects = true 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.432 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore compression algorithm = zstd 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore compression mode = aggressive 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore zero block detection = True 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 20 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 20 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 20 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-29T10:45:26.433 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2024-03-29T10:45:26.434 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 10 2024-03-29T10:45:26.434 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:45:26.434 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2024-03-29T10:45:26.434 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:45:26.434 INFO:teuthology.orchestra.run.smithi174.stdout: osd class default list = * 2024-03-29T10:45:26.434 INFO:teuthology.orchestra.run.smithi174.stdout: osd class load list = * 2024-03-29T10:45:26.435 INFO:teuthology.orchestra.run.smithi174.stdout: osd client watch timeout = 120 2024-03-29T10:45:26.435 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify cached snaps = True 2024-03-29T10:45:26.435 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify missing on start = True 2024-03-29T10:45:26.435 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:45:26.435 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock override recovery settings = True 2024-03-29T10:45:26.435 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2024-03-29T10:45:26.436 INFO:teuthology.orchestra.run.smithi174.stdout: osd op queue = debug_random 2024-03-29T10:45:26.436 INFO:teuthology.orchestra.run.smithi174.stdout: osd op queue cut off = debug_random 2024-03-29T10:45:26.436 INFO:teuthology.orchestra.run.smithi174.stdout: osd shutdown pgref assert = True 2024-03-29T10:45:26.436 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.436 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2024-03-29T10:45:26.436 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-29T10:45:26.437 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2024-03-29T10:45:26.437 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-03-29T10:45:26.437 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-03-29T10:45:26.437 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:45:26.437 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:45:26.437 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:45:26.438 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.439 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon min osdmap epochs = 50 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon osdmap full prune interval = 2 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon osdmap full prune min = 15 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon scrub interval = 300 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on pool no app = False 2024-03-29T10:45:26.440 INFO:teuthology.orchestra.run.smithi174.stdout: paxos service trim min = 10 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 20 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: debug objecter = 20 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout: debug rados = 20 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2024-03-29T10:45:26.441 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2024-03-29T10:45:26.443 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:45:26.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:45:26.498 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:45:26.498 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:26.498 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:45:26.513 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:26.513 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:45:26.572 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:45:26.572 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:26.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:45:26.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:45:26.651 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:26.651 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:26.706 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:45:26.706 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:26.706 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:45:26.706 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:45:26.746 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:26.746 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:26.802 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:45:26.802 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:45:26.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:45:26.857 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:45:26.860 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:45:26.923 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:45:26.926 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:45:26.926 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:45:26.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:45:26.968 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:45:26.984 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:45:27.027 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:45:27.042 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:45:27.042 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:45:27.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:45:27.069 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:27.069 INFO:tasks.ceph:role: osd.0 2024-03-29T10:45:27.069 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:45:27.069 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:45:27.143 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:27.143 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:27.144 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:27.660 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T10:45:27.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T10:45:27.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:45:27.788 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:45:27.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:45:27.887 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:27.887 INFO:tasks.ceph:role: osd.1 2024-03-29T10:45:27.887 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:45:27.888 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:45:27.961 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:27.962 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:27.962 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:27.962 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:27.962 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:28.418 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T10:45:28.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T10:45:28.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:45:28.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:45:28.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:45:28.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:28.609 INFO:tasks.ceph:role: osd.2 2024-03-29T10:45:28.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:45:28.609 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:28.684 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:29.180 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T10:45:29.182 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T10:45:29.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:45:29.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:45:29.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:45:29.403 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:29.403 INFO:tasks.ceph:role: osd.3 2024-03-29T10:45:29.403 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:45:29.403 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:29.477 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:30.001 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-29T10:45:30.003 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-29T10:45:30.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:45:30.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:45:30.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:30.280+0000 7f93ffe0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:45:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:30.280+0000 7f93ffe0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:45:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:30.280+0000 7f93ffe0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:45:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:30.280+0000 7f93ffe0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:45:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:30.280+0000 7f93ffe0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:45:30.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:30.280+0000 7f93ffe0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:45:33.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:45:33.152 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:33.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:33.262+0000 7f2e4cc01640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:45:33.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:33.262+0000 7f2e4cc01640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:45:33.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:33.262+0000 7f2e4cc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:45:33.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:33.262+0000 7f2e4cc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:45:33.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:33.262+0000 7f2e4cc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:45:33.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:33.263+0000 7f2e4cc01640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:45:36.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:45:36.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:36.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:36.233+0000 7feca5d93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:45:36.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:36.234+0000 7feca5d93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:45:36.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:36.234+0000 7feca5d93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:45:36.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:36.234+0000 7feca5d93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:45:36.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:36.234+0000 7feca5d93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:45:36.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:36.234+0000 7feca5d93640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:45:39.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:45:39.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:39.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:39.218+0000 7f4573613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:45:39.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:39.218+0000 7f4573613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:45:39.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:39.218+0000 7f4573613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:45:39.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:39.218+0000 7f4573613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:45:39.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:39.218+0000 7f4573613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:45:39.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:39.219+0000 7f4573613640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:45:42.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:45:42.083 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:45:42.116 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:42.116 INFO:tasks.ceph:role: osd.4 2024-03-29T10:45:42.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:45:42.116 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:42.194 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:42.195 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:42.195 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:42.195 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:42.693 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-29T10:45:42.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-29T10:45:42.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:45:42.806 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:45:42.847 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:45:42.915 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:42.915 INFO:tasks.ceph:role: osd.5 2024-03-29T10:45:42.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:45:42.915 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:42.990 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:42.991 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:43.450 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-29T10:45:43.452 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-29T10:45:43.452 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:45:43.543 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:45:43.573 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:45:43.641 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:43.641 INFO:tasks.ceph:role: osd.6 2024-03-29T10:45:43.641 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:45:43.641 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:45:43.716 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:43.716 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:43.716 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:43.716 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:43.716 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:43.716 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:43.717 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:43.717 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:43.717 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:43.717 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:44.208 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-29T10:45:44.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-29T10:45:44.211 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:45:44.337 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:45:44.370 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:45:44.439 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:45:44.439 INFO:tasks.ceph:role: osd.7 2024-03-29T10:45:44.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:45:44.440 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:45:44.514 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:45:44.515 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:45:45.020 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-03-29T10:45:45.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-03-29T10:45:45.023 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:45:45.180 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:45:45.214 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:45.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:45.325+0000 7ff17982a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:45:45.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:45.325+0000 7ff17982a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:45:45.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:45.325+0000 7ff17982a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:45:45.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:45.325+0000 7ff17982a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:45:45.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:45.325+0000 7ff17982a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:45:45.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:45.326+0000 7ff17982a640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:45:48.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:45:48.190 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:48.288 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:48.304+0000 7f60f260c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:45:48.289 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:48.304+0000 7f60f260c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:45:48.289 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:48.304+0000 7f60f260c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:45:48.289 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:48.304+0000 7f60f260c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:45:48.289 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:48.304+0000 7f60f260c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:45:48.289 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:48.305+0000 7f60f260c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:45:51.146 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:45:51.179 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:51.273 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:51.290+0000 7f28500a4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:45:51.273 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:51.290+0000 7f28500a4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:45:51.274 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:51.290+0000 7f28500a4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:45:51.274 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:51.290+0000 7f28500a4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:45:51.274 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:51.290+0000 7f28500a4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:45:51.274 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:51.291+0000 7f28500a4640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:45:54.126 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:45:54.157 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:54.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:54.267+0000 7f1346a48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:45:54.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:54.267+0000 7f1346a48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:45:54.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:54.267+0000 7f1346a48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:45:54.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:54.267+0000 7f1346a48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:45:54.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:54.267+0000 7f1346a48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:45:54.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:45:54.268+0000 7f1346a48640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:45:57.102 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:45:57.133 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:45:57.133 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:57.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:45:57.166 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:57.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:45:57.236 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:57.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:45:57.306 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:57.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:45:57.375 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:57.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:45:57.444 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:57.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:45:57.476 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:57.476 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:45:57.546 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:57.546 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:45:57.616 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:57.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:45:57.685 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:57.685 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:45:57.754 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:45:57.754 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:45:57.771 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:45:57.771 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:45:57.809 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:45:57.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:45:57.813 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:45:57.852 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-03-29T10:45:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDGmwZmXGgzNBAAYiq5mFUTm156NkLHQmCA9Q== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDKmwZmH5a+EBAAjp1S9AGsnoJ+X8KeRFzo0A== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDNmwZm0TmvDxAAwZjCn1aTOWK+IuXEDJ+DTg== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDQmwZmf3T+DRAAJnR72/R5e5ieK+pxbjtqsA== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDTmwZmUfgRDRAAUSUFZWupesCc7UG5tp2rvA== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDGmwZmu+wmOBAAsa5Z57ICzGRBGyO0CLa8GQ== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDZmwZmUUJ0ExAA05FRbBRFAC+4U63lIKS5Ng== 2024-03-29T10:45:57.853 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDcmwZm5u0tEhAAGFe05vUtLJMTccCz43lfQA== 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDfmwZm+DNhERAAezqzXvzO14HEA7qBa+xsaQ== 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDimwZmKyP9DxAA2sqhrSgGW+vmeb5pNSGXMA== 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDGmwZmrSbZOhAAKzlwqdaen5QsjDCJkDbPaA== 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-03-29T10:45:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDHmwZmj4KzAhAABTw3+pw8DgOmUqgW9D6eYg== 2024-03-29T10:45:57.876 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.y] 2024-03-29T10:45:57.876 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDGmwZmXGgzNBAAYiq5mFUTm156NkLHQmCA9Q== 2024-03-29T10:45:57.876 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-03-29T10:45:57.876 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDKmwZmH5a+EBAAjp1S9AGsnoJ+X8KeRFzo0A== 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDNmwZm0TmvDxAAwZjCn1aTOWK+IuXEDJ+DTg== 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDQmwZmf3T+DRAAJnR72/R5e5ieK+pxbjtqsA== 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDTmwZmUfgRDRAAUSUFZWupesCc7UG5tp2rvA== 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDGmwZmu+wmOBAAsa5Z57ICzGRBGyO0CLa8GQ== 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.4] 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDZmwZmUUJ0ExAA05FRbBRFAC+4U63lIKS5Ng== 2024-03-29T10:45:57.877 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.5] 2024-03-29T10:45:57.878 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDcmwZm5u0tEhAAGFe05vUtLJMTccCz43lfQA== 2024-03-29T10:45:57.878 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.6] 2024-03-29T10:45:57.878 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDfmwZm+DNhERAAezqzXvzO14HEA7qBa+xsaQ== 2024-03-29T10:45:57.878 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.7] 2024-03-29T10:45:57.878 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDimwZmKyP9DxAA2sqhrSgGW+vmeb5pNSGXMA== 2024-03-29T10:45:57.878 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-03-29T10:45:57.879 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDGmwZmrSbZOhAAKzlwqdaen5QsjDCJkDbPaA== 2024-03-29T10:45:57.879 INFO:teuthology.orchestra.run.smithi174.stdout:[client.1] 2024-03-29T10:45:57.879 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDHmwZmj4KzAhAABTw3+pw8DgOmUqgW9D6eYg== 2024-03-29T10:45:57.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:45:57.898 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:45:57.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:57.976 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.030 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.084 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.137 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:45:58.191 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:45:58.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.244 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.298 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.405 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:45:58.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:45:58.458 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:45:58.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:45:58.516 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:45:58.571 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:45:58.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:45:58.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:45:58.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:45:58.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:45:58.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:45:58.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:45:58.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:45:58.936 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:45:59.033 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:45:59.059 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:59.062 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:45:59.114 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:45:59.114 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:45:59.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:45:59.116 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:45:59.117 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:45:59.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:45:59.119 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:45:59.119 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:45:59.119 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:45:59.156 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:45:59.157 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:45:59.157 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:45:59.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:45:59.159 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:45:59.159 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:45:59.159 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:45:59.161 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:45:59.161 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:45:59.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.281+0000 7fcf41579640 10 monclient: get_monmap_and_config 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.281+0000 7fcf41579640 10 monclient: build_initial_monmap 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.281+0000 7fcf41579640 10 monclient: monmap: 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:59.283445+0000 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:59.283445+0000 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:59.265 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:45:59.266 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:45:59.266 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:45:59.266 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:45:59.266 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:45:59.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 1 Processor -- start 2024-03-29T10:45:59.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 1 -- start start 2024-03-29T10:45:59.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 10 monclient: init 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c0ae3c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c0aae90 addr v1:172.21.15.64:6789/0 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c071dc0 addr v1:172.21.15.64:6790/0 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.282+0000 7fcf41579640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ac7f0 con 0x7fcf3c0aae90 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf41579640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ab370 con 0x7fcf3c0ae3c0 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf41579640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ab890 con 0x7fcf3c071dc0 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf41579640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf41579640 10 monclient(hunting): authenticate will time out at 770.582880s 2024-03-29T10:45:59.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf33fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcf3c0aae90 0x7fcf3c0acef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47466/0 (socket says 172.21.15.64:47466) 2024-03-29T10:45:59.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf33fff640 1 -- 172.21.15.64:0/836826419 learned_addr learned my addr 172.21.15.64:0/836826419 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:45:59.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:45:59.283+0000 7fcf3b7fe640 1 --1- 172.21.15.64:0/836826419 >> v1:172.21.15.64:6790/0 conn(0x7fcf3c071dc0 0x7fcf3c14a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:57332/0 (socket says 172.21.15.64:57332) 2024-03-29T10:46:02.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): tick 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 monclient(hunting): continuing hunt 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 -- 172.21.15.64:0/836826419 >> v1:172.21.15.64:6790/0 conn(0x7fcf3c071dc0 legacy=0x7fcf3c14a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 -- 172.21.15.64:0/836826419 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 legacy=0x7fcf3c13c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 -- 172.21.15.64:0/836826419 >> v1:172.21.15.64:6789/0 conn(0x7fcf3c0aae90 legacy=0x7fcf3c0acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c0058f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c0ae3c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c071dc0 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 -- 172.21.15.64:0/836826419 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ad600 con 0x7fcf1c0058f0 2024-03-29T10:46:02.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 -- 172.21.15.64:0/836826419 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c011590 con 0x7fcf3c0ae3c0 2024-03-29T10:46:02.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 1 -- 172.21.15.64:0/836826419 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0127f0 con 0x7fcf3c071dc0 2024-03-29T10:46:02.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:02.283+0000 7fcf397fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:04.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.449+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779847346 0 0) 0x7fcf3c0ad600 con 0x7fcf1c0058f0 2024-03-29T10:46:04.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.449+0000 7fcf3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:04.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.449+0000 7fcf3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:04.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.449+0000 7fcf3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:04.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.449+0000 7fcf3a7fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:46:04.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.449+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf28003520 con 0x7fcf1c0058f0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460463735 0 0) 0x7fcf28003520 con 0x7fcf1c0058f0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 >> v1:172.21.15.64:6790/0 conn(0x7fcf3c071dc0 legacy=0x7fcf1c00ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 legacy=0x7fcf1c00b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.451+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c013a50 con 0x7fcf1c0058f0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcf2c0016d0 con 0x7fcf1c0058f0 2024-03-29T10:46:04.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: dump: 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf41579640 5 monclient: authenticate success, global_id 4101 2024-03-29T10:46:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf41579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf2c001920 con 0x7fcf1c0058f0 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 1 -- 172.21.15.64:0/836826419 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcf2c001cb0 con 0x7fcf1c0058f0 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf41579640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf41579640 4 set_mon_vals no callback set 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.452+0000 7fcf3a7fc640 10 monclient: dump: 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:04.439 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:04.439 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:04.439 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:04.439 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 10 monclient: shutdown 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 1 -- 172.21.15.64:0/836826419 >> v1:172.21.15.64:6789/0 conn(0x7fcf1c0058f0 legacy=0x7fcf1c007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 1 -- 172.21.15.64:0/836826419 shutdown_connections 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 1 -- 172.21.15.64:0/836826419 >> 172.21.15.64:0/836826419 conn(0x7fcf3c0b3750 msgr2=0x7fcf3c0b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 1 -- 172.21.15.64:0/836826419 shutdown_connections 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.453+0000 7fcf41579640 1 -- 172.21.15.64:0/836826419 wait complete. 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.454+0000 7fcf41579640 10 monclient: build_initial_monmap 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.454+0000 7fcf41579640 10 monclient: monmap: 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:04.444 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.454+0000 7fcf41579640 1 librados: starting msgr at 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.454+0000 7fcf41579640 1 librados: starting objecter 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.454+0000 7fcf41579640 1 Processor -- start 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 1 -- start start 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 1 librados: setting wanted keys 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 1 librados: calling monclient init 2024-03-29T10:46:04.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient: init 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c0058f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c071dc0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c0ae3c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ada00 con 0x7fcf3c071dc0 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c11c300 con 0x7fcf3c0ae3c0 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c135a70 con 0x7fcf1c0058f0 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf41579640 10 monclient(hunting): authenticate will time out at 775.755732s 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf3affd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcf1c0058f0 0x7fcf3c1daca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35688/0 (socket says 172.21.15.64:35688) 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.455+0000 7fcf3affd640 1 -- 172.21.15.64:0/2279281455 learned_addr learned my addr 172.21.15.64:0/2279281455 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:04.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 client.?.objecter ms_handle_connect 0x7fcf1c0058f0 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf3b7fe640 1 -- 172.21.15.64:0/2279281455 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 legacy=0x7fcf3c142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf3b7fe640 1 -- 172.21.15.64:0/2279281455 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 legacy=0x7fcf3c142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf3b7fe640 1 --1- 172.21.15.64:0/2279281455 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 0x7fcf3c142430 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf3b7fe640 1 --1- 172.21.15.64:0/2279281455 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 0x7fcf3c142430 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 client.?.objecter ms_handle_connect 0x7fcf3c071dc0 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.174:6789/0 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553594088 0 0) 0x7fcf3c0ada00 con 0x7fcf3c071dc0 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:46:04.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.456+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c013220 con 0x7fcf3c071dc0 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3405810220 0 0) 0x7fcf1c013220 con 0x7fcf3c071dc0 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf3c0ada00 con 0x7fcf3c071dc0 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcf24001720 con 0x7fcf3c071dc0 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient(hunting): dump: 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:04.448 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3543618424 0 0) 0x7fcf3c0ada00 con 0x7fcf3c071dc0 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 >> v1:172.21.15.64:6790/0 conn(0x7fcf1c0058f0 legacy=0x7fcf3c1daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 >> v1:172.21.15.174:6789/0 conn(0x7fcf3c0ae3c0 legacy=0x7fcf3c142430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 monclient: found mon.a 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c135c00 con 0x7fcf3c071dc0 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:04.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 10 monclient: _renew_subs 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3c1e2f80 con 0x7fcf3c071dc0 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 10 monclient: _renew_subs 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c1e32f0 con 0x7fcf3c071dc0 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf41579640 1 librados: init done 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf24001cb0 con 0x7fcf3c071dc0 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcf24004980 con 0x7fcf3c071dc0 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:04.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf39ffb640 4 set_mon_vals no callback set 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 monclient: dump: 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcf24004e70 con 0x7fcf3c071dc0 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2480178181 0 0) 0x7fcf24006440 con 0x7fcf3c071dc0 2024-03-29T10:46:04.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 10 client.4113.objecter ms_dispatch 0x7fcf3c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.457+0000 7fcf337fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.458+0000 7fcf337fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.458+0000 7fcf337fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.458+0000 7fcf41579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.458+0000 7fcf41579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.458+0000 7fcf41579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.458+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf20005080 con 0x7fcf3c071dc0 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.461+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcf24006650 con 0x7fcf3c071dc0 2024-03-29T10:46:04.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.461+0000 7fcf337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:04.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.461+0000 7fcf337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.534+0000 7fcf41579640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:46:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.534+0000 7fcf41579640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:46:04.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.534+0000 7fcf41579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:04.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.534+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcf20005330 con 0x7fcf3c071dc0 2024-03-29T10:46:04.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.536+0000 7fcf337fe640 1 -- 172.21.15.64:0/2279281455 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fcf240053e0 con 0x7fcf3c071dc0 2024-03-29T10:46:04.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.536+0000 7fcf337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:46:04.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.536+0000 7fcf337fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:46:04.519 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-03-29T10:46:04.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.536+0000 7fcf41579640 10 librados: watch_flush enter 2024-03-29T10:46:04.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 10 librados: watch_flush exit 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 10 monclient: shutdown 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 >> v1:172.21.15.64:6789/0 conn(0x7fcf3c071dc0 legacy=0x7fcf3c141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 shutdown_connections 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 >> 172.21.15.64:0/2279281455 conn(0x7fcf3c0b3750 msgr2=0x7fcf3c0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 shutdown_connections 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 1 -- 172.21.15.64:0/2279281455 wait complete. 2024-03-29T10:46:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:04.537+0000 7fcf41579640 1 librados: shutdown 2024-03-29T10:46:04.532 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:46:04.532 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:46:04.532 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:46:04.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:46:04.559 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:46:04.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:46:04.625 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:46:04.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:46:04.692 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-29T10:46:04.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:46:04.759 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:46:04.759 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:46:04.786 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:46:04.786 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:46:04.853 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:46:04.853 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:46:04.920 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-29T10:46:04.921 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:46:04.926 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:04.943+0000 7f589663e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:46:04.937 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:04.955+0000 7f0fafa3c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:46:04.964 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 29d8829f-a8b5-4888-a248-7d3f3ad95394 0 2024-03-29T10:46:04.983 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:04.999+0000 7f589663e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:46:04.994 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.012+0000 7f0fafa3c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:46:05.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.110+0000 7f92ee831640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.110+0000 7f92ee831640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.110+0000 7f92ee831640 10 monclient: monmap: 2024-03-29T10:46:05.093 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:05.111557+0000 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:05.111557+0000 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:05.094 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.111+0000 7f92ee831640 1 Processor -- start 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.111+0000 7f92ee831640 1 -- start start 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.111+0000 7f92ee831640 10 monclient: init 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.111+0000 7f92ee831640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.111+0000 7f92ee831640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.111+0000 7f92ee831640 10 monclient(hunting): picked mon.noname-b con 0x7f92e809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 10 monclient(hunting): picked mon.noname-a con 0x7f92e809c870 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 10 monclient(hunting): picked mon.noname-c con 0x7f92e80b4360 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e809dec0 con 0x7f92e809c870 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e809da30 con 0x7f92e80b4360 2024-03-29T10:46:05.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e80b39c0 con 0x7f92e809ac40 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 10 monclient(hunting): authenticate will time out at 760.388016s 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ecdce640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f92e80b4360 0x7f92e814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60456/0 (socket says 172.21.15.174:60456) 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ecdce640 1 -- 172.21.15.174:0/224931240 learned_addr learned my addr 172.21.15.174:0/224931240 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650737128 0 0) 0x7f92e809da30 con 0x7f92e80b4360 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0003520 con 0x7f92e80b4360 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317627000 0 0) 0x7f92d8001110 con 0x7f92e809c870 2024-03-29T10:46:05.108 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e809da30 con 0x7f92e809c870 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676229278 0 0) 0x7f92e80b39c0 con 0x7f92e809ac40 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.109 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d8001110 con 0x7f92e809ac40 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4112514243 0 0) 0x7f92d0003520 con 0x7f92e80b4360 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 >> v1:172.21.15.64:6790/0 conn(0x7f92e809ac40 legacy=0x7f92e80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 >> v1:172.21.15.64:6789/0 conn(0x7f92e809c870 legacy=0x7f92e813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 monclient: found mon.noname-c 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80b5160 con 0x7f92e80b4360 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f92e0001a10 con 0x7f92e80b4360 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: dump: 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.111 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e0001bf0 con 0x7f92e80b4360 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 1 -- 172.21.15.174:0/224931240 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f92e0002000 con 0x7f92e80b4360 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92e6ffd640 10 monclient: dump: 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.113 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.112+0000 7f92ee831640 4 set_mon_vals no callback set 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 10 monclient: shutdown 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 1 -- 172.21.15.174:0/224931240 >> v1:172.21.15.174:6789/0 conn(0x7f92e80b4360 legacy=0x7f92e814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 1 -- 172.21.15.174:0/224931240 shutdown_connections 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 1 -- 172.21.15.174:0/224931240 >> 172.21.15.174:0/224931240 conn(0x7f92e8066b00 msgr2=0x7f92e8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 1 -- 172.21.15.174:0/224931240 shutdown_connections 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.113+0000 7f92ee831640 1 -- 172.21.15.174:0/224931240 wait complete. 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.114+0000 7f92ee831640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.114+0000 7f92ee831640 10 monclient: monmap: 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.114+0000 7f92ee831640 1 librados: starting msgr at 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.114+0000 7f92ee831640 1 librados: starting objecter 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.115+0000 7f92ee831640 1 Processor -- start 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.115+0000 7f92ee831640 1 -- start start 2024-03-29T10:46:05.115 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.115+0000 7f92ee831640 1 librados: setting wanted keys 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.115+0000 7f92ee831640 1 librados: calling monclient init 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.115+0000 7f92ee831640 10 monclient: init 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient(hunting): picked mon.noname-c con 0x7f92e809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient(hunting): picked mon.noname-a con 0x7f92e81d7cd0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient(hunting): picked mon.noname-b con 0x7f92e80ad1e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e81d7130 con 0x7f92e81d7cd0 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e8139b10 con 0x7f92e80ad1e0 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e813ffa0 con 0x7f92e809ac40 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ee831640 10 monclient(hunting): authenticate will time out at 760.392207s 2024-03-29T10:46:05.116 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ecdce640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f92e80ad1e0 0x7f92e8139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60462/0 (socket says 172.21.15.174:60462) 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92ecdce640 1 -- 172.21.15.174:0/2826782181 learned_addr learned my addr 172.21.15.174:0/2826782181 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 client.?.objecter ms_handle_connect 0x7f92e80ad1e0 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 client.?.objecter ms_handle_connect 0x7f92e809ac40 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 client.?.objecter ms_handle_connect 0x7f92e81d7cd0 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336641728 0 0) 0x7f92e8139b10 con 0x7f92e80ad1e0 2024-03-29T10:46:05.117 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4004eb0 con 0x7f92e80ad1e0 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778643376 0 0) 0x7f92e813ffa0 con 0x7f92e809ac40 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8139b10 con 0x7f92e809ac40 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228535894 0 0) 0x7f92e81d7130 con 0x7f92e81d7cd0 2024-03-29T10:46:05.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e813ffa0 con 0x7f92e81d7cd0 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2880754872 0 0) 0x7f92d4004eb0 con 0x7f92e80ad1e0 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e81d7130 con 0x7f92e80ad1e0 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f92e0001f40 con 0x7f92e80ad1e0 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.119 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069577289 0 0) 0x7f92e813ffa0 con 0x7f92e81d7cd0 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d4004eb0 con 0x7f92e81d7cd0 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f92d80018a0 con 0x7f92e81d7cd0 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:05.120 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.116+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3286170432 0 0) 0x7f92e81d7130 con 0x7f92e80ad1e0 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 >> v1:172.21.15.64:6790/0 conn(0x7f92e809ac40 legacy=0x7f92e81d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 >> v1:172.21.15.64:6789/0 conn(0x7f92e81d7cd0 legacy=0x7f92e8138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 monclient: found mon.b 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80ae520 con 0x7f92e80ad1e0 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 10 monclient: _renew_subs 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e80ae800 con 0x7f92e80ad1e0 2024-03-29T10:46:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 10 monclient: _renew_subs 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e8142b70 con 0x7f92e80ad1e0 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e0004e10 con 0x7f92e80ad1e0 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 1 librados: init done 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f92e0005120 con 0x7f92e80ad1e0 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 monclient: dump: 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e67fc640 4 set_mon_vals no callback set 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f92e0005610 con 0x7f92e80ad1e0 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 4067476371 0 0) 0x7f92e0006ae0 con 0x7f92e80ad1e0 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 10 client.4111.objecter ms_dispatch 0x7f92e8000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:46:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92e4ff9640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.117+0000 7f92ee831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.118+0000 7f92ee831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.118+0000 7f92ee831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.118+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e81d80b0 con 0x7f92e80ad1e0 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.120+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f92e0005b50 con 0x7f92e80ad1e0 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.120+0000 7f92e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.120+0000 7f92e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.174 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.191+0000 7f92ee831640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "29d8829f-a8b5-4888-a248-7d3f3ad95394", "id": 0}] 2024-03-29T10:46:05.174 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.191+0000 7f92ee831640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "29d8829f-a8b5-4888-a248-7d3f3ad95394", "id": 0}] 2024-03-29T10:46:05.174 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.191+0000 7f92ee831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.174 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.191+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "29d8829f-a8b5-4888-a248-7d3f3ad95394", "id": 0} v 0) v1 -- 0x7f92e8142ec0 con 0x7f92e80ad1e0 2024-03-29T10:46:05.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.197+0000 7f92e4ff9640 1 -- 172.21.15.174:0/2826782181 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29d8829f-a8b5-4888-a248-7d3f3ad95394", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 4060601561 0 732707515) 0x7f92e0005dc0 con 0x7f92e80ad1e0 2024-03-29T10:46:05.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.197+0000 7f92e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "29d8829f-a8b5-4888-a248-7d3f3ad95394", "id": 0}] 2024-03-29T10:46:05.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.197+0000 7f92e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:05.180 INFO:teuthology.orchestra.run.smithi174.stdout:0 2024-03-29T10:46:05.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 10 librados: watch_flush enter 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 10 librados: watch_flush exit 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 10 monclient: shutdown 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 >> v1:172.21.15.174:6789/0 conn(0x7f92e80ad1e0 legacy=0x7f92e8139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 shutdown_connections 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 >> 172.21.15.174:0/2826782181 conn(0x7f92e8066b00 msgr2=0x7f92e80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 shutdown_connections 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 1 -- 172.21.15.174:0/2826782181 wait complete. 2024-03-29T10:46:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.198+0000 7f92ee831640 1 librados: shutdown 2024-03-29T10:46:05.195 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new b58ab0d3-43de-4f0c-b2fb-a6a557729219 1 2024-03-29T10:46:05.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.311+0000 7f6d9731c640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.311+0000 7f6d9731c640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.311+0000 7f6d9731c640 10 monclient: monmap: 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:05.312700+0000 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:05.312700+0000 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.296 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:05.296 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:05.296 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.296 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.312+0000 7f6d9731c640 1 Processor -- start 2024-03-29T10:46:05.296 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.312+0000 7f6d9731c640 1 -- start start 2024-03-29T10:46:05.296 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.312+0000 7f6d9731c640 10 monclient: init 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9009ac40 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9009c870 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient(hunting): picked mon.noname-c con 0x7f6d900b4360 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9009dec0 con 0x7f6d9009ac40 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9009da30 con 0x7f6d900b4360 2024-03-29T10:46:05.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d900b39c0 con 0x7f6d9009c870 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient(hunting): authenticate will time out at 760.589148s 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d958b9640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6d900b4360 0x7f6d9014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60476/0 (socket says 172.21.15.174:60476) 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d958b9640 1 -- 172.21.15.174:0/96562005 learned_addr learned my addr 172.21.15.174:0/96562005 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445220197 0 0) 0x7f6d9009da30 con 0x7f6d900b4360 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c003520 con 0x7f6d900b4360 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287199656 0 0) 0x7f6d900b39c0 con 0x7f6d9009c870 2024-03-29T10:46:05.306 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9009da30 con 0x7f6d9009c870 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306424440 0 0) 0x7f6d80001110 con 0x7f6d9009ac40 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d900b39c0 con 0x7f6d9009ac40 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291026152 0 0) 0x7f6d7c003520 con 0x7f6d900b4360 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 >> v1:172.21.15.64:6790/0 conn(0x7f6d9009c870 legacy=0x7f6d9013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 >> v1:172.21.15.64:6789/0 conn(0x7f6d9009ac40 legacy=0x7f6d900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.307 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 monclient: found mon.noname-c 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900b5160 con 0x7f6d900b4360 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6d88001a10 con 0x7f6d900b4360 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: dump: 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d88001bf0 con 0x7f6d900b4360 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 1 -- 172.21.15.174:0/96562005 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6d88002000 con 0x7f6d900b4360 2024-03-29T10:46:05.309 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d9731c640 4 set_mon_vals no callback set 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.313+0000 7f6d87fff640 10 monclient: dump: 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.310 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 10 monclient: shutdown 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 1 -- 172.21.15.174:0/96562005 >> v1:172.21.15.174:6789/0 conn(0x7f6d900b4360 legacy=0x7f6d9014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 1 -- 172.21.15.174:0/96562005 shutdown_connections 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 1 -- 172.21.15.174:0/96562005 >> 172.21.15.174:0/96562005 conn(0x7f6d90066b00 msgr2=0x7f6d90066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 1 -- 172.21.15.174:0/96562005 shutdown_connections 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.314+0000 7f6d9731c640 1 -- 172.21.15.174:0/96562005 wait complete. 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 10 monclient: monmap: 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.311 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 1 librados: starting msgr at 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 1 librados: starting objecter 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 1 Processor -- start 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 1 -- start start 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 1 librados: setting wanted keys 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 1 librados: calling monclient init 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.315+0000 7f6d9731c640 10 monclient: init 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:05.312 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9009ac40 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient(hunting): picked mon.noname-a con 0x7f6d90144760 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient(hunting): picked mon.noname-c con 0x7f6d90136e60 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d90142b60 con 0x7f6d90144760 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d90137950 con 0x7f6d9009ac40 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d90142dd0 con 0x7f6d90136e60 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d950b8640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6d9009ac40 0x7f6d90140c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60478/0 (socket says 172.21.15.174:60478) 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient(hunting): authenticate will time out at 760.592186s 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d950b8640 1 -- 172.21.15.174:0/2837575158 learned_addr learned my addr 172.21.15.174:0/2837575158 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 client.?.objecter ms_handle_connect 0x7f6d9009ac40 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 client.?.objecter ms_handle_connect 0x7f6d90144760 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 client.?.objecter ms_handle_connect 0x7f6d90136e60 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105365922 0 0) 0x7f6d90137950 con 0x7f6d9009ac40 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74004eb0 con 0x7f6d9009ac40 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258339439 0 0) 0x7f6d90142b60 con 0x7f6d90144760 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:46:05.314 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d90137950 con 0x7f6d90144760 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834772273 0 0) 0x7f6d90142dd0 con 0x7f6d90136e60 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d90142b60 con 0x7f6d90136e60 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1856870319 0 0) 0x7f6d74004eb0 con 0x7f6d9009ac40 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d90142dd0 con 0x7f6d9009ac40 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6d80001860 con 0x7f6d9009ac40 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.315 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4161392893 0 0) 0x7f6d90137950 con 0x7f6d90144760 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74004eb0 con 0x7f6d90144760 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6d8c0017e0 con 0x7f6d90144760 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 345370497 0 0) 0x7f6d90142b60 con 0x7f6d90136e60 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d90137950 con 0x7f6d90136e60 2024-03-29T10:46:05.317 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6d88001f00 con 0x7f6d90136e60 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.318 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 253110624 0 0) 0x7f6d90142dd0 con 0x7f6d9009ac40 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 >> v1:172.21.15.64:6790/0 conn(0x7f6d90136e60 legacy=0x7f6d90137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 >> v1:172.21.15.64:6789/0 conn(0x7f6d90144760 legacy=0x7f6d90144bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.319 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 monclient: found mon.b 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d901430b0 con 0x7f6d9009ac40 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient: _renew_subs 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.316+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d90141900 con 0x7f6d9009ac40 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d80003610 con 0x7f6d9009ac40 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6d80003a60 con 0x7f6d9009ac40 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 10 monclient: dump: 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.321 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d877fe640 4 set_mon_vals no callback set 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d9731c640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d9731c640 10 monclient: _renew_subs 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d9731c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d90141e00 con 0x7f6d9009ac40 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6d80003fd0 con 0x7f6d9009ac40 2024-03-29T10:46:05.322 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d9731c640 1 librados: init done 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3670796427 0 0) 0x7f6d80005b70 con 0x7f6d9009ac40 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 10 client.4117.objecter ms_dispatch 0x7f6d90000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.317+0000 7f6d85ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.318+0000 7f6d9731c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.318+0000 7f6d9731c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.318+0000 7f6d9731c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.318+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9009b020 con 0x7f6d9009ac40 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.319+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6d80004280 con 0x7f6d9009ac40 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.319+0000 7f6d85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.323 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.319+0000 7f6d85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.391+0000 7f6d9731c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b58ab0d3-43de-4f0c-b2fb-a6a557729219", "id": 1}] 2024-03-29T10:46:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.391+0000 7f6d9731c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b58ab0d3-43de-4f0c-b2fb-a6a557729219", "id": 1}] 2024-03-29T10:46:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.391+0000 7f6d9731c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.391+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b58ab0d3-43de-4f0c-b2fb-a6a557729219", "id": 1} v 0) v1 -- 0x7f6d9004f730 con 0x7f6d9009ac40 2024-03-29T10:46:05.381 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d85ffb640 1 -- 172.21.15.174:0/2837575158 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b58ab0d3-43de-4f0c-b2fb-a6a557729219", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1957717246 0 940483276) 0x7f6d80004420 con 0x7f6d9009ac40 2024-03-29T10:46:05.381 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b58ab0d3-43de-4f0c-b2fb-a6a557729219", "id": 1}] 2024-03-29T10:46:05.381 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:05.381 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 10 librados: watch_flush enter 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 10 librados: watch_flush exit 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 10 monclient: shutdown 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.399+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 >> v1:172.21.15.174:6789/0 conn(0x7f6d9009ac40 legacy=0x7f6d90140c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.400+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 shutdown_connections 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.400+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 >> 172.21.15.174:0/2837575158 conn(0x7f6d90066b00 msgr2=0x7f6d9013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.400+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 shutdown_connections 2024-03-29T10:46:05.383 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.400+0000 7f6d9731c640 1 -- 172.21.15.174:0/2837575158 wait complete. 2024-03-29T10:46:05.384 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.400+0000 7f6d9731c640 1 librados: shutdown 2024-03-29T10:46:05.392 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new ab146959-79eb-4128-b51c-c83a6258a4fa 2 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.510+0000 7f434f8e6640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.510+0000 7f434f8e6640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.510+0000 7f434f8e6640 10 monclient: monmap: 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:05.511253+0000 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:05.511253+0000 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:05.493 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:05.494 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.494 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.510+0000 7f434f8e6640 1 Processor -- start 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 1 -- start start 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient: init 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f434809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f434809c870 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f43480b4360 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434809dec0 con 0x7f43480b4360 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434809da30 con 0x7f434809c870 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480b39c0 con 0x7f434809ac40 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434f8e6640 10 monclient(hunting): authenticate will time out at 760.787613s 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434ce81640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f434809c870 0x7f434813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60488/0 (socket says 172.21.15.174:60488) 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.511+0000 7f434ce81640 1 -- 172.21.15.174:0/1874137824 learned_addr learned my addr 172.21.15.174:0/1874137824 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883946828 0 0) 0x7f434809da30 con 0x7f434809c870 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4334003520 con 0x7f434809c870 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106591399 0 0) 0x7f43480b39c0 con 0x7f434809ac40 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f434809da30 con 0x7f434809ac40 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076619613 0 0) 0x7f4334003520 con 0x7f434809c870 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 >> v1:172.21.15.64:6790/0 conn(0x7f434809ac40 legacy=0x7f43480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 >> v1:172.21.15.64:6789/0 conn(0x7f43480b4360 legacy=0x7f434814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 monclient: found mon.noname-c 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43480b5160 con 0x7f434809c870 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.505 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4338001690 con 0x7f434809c870 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: dump: 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.506 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4338001870 con 0x7f434809c870 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 1 -- 172.21.15.174:0/1874137824 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4338001c80 con 0x7f434809c870 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 4 set_mon_vals no callback set 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f433ffff640 10 monclient: dump: 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.508 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 10 monclient: shutdown 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.512+0000 7f434f8e6640 1 -- 172.21.15.174:0/1874137824 >> v1:172.21.15.174:6789/0 conn(0x7f434809c870 legacy=0x7f434813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 1 -- 172.21.15.174:0/1874137824 shutdown_connections 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 1 -- 172.21.15.174:0/1874137824 >> 172.21.15.174:0/1874137824 conn(0x7f4348066b00 msgr2=0x7f4348066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 1 -- 172.21.15.174:0/1874137824 shutdown_connections 2024-03-29T10:46:05.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 1 -- 172.21.15.174:0/1874137824 wait complete. 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 10 monclient: monmap: 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.510 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 1 librados: starting msgr at 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.513+0000 7f434f8e6640 1 librados: starting objecter 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 Processor -- start 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 -- start start 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 librados: setting wanted keys 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 librados: calling monclient init 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient: init 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f434809ac40 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f43480b6b00 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f43481406b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43481d7e20 con 0x7f434809ac40 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4348140b00 con 0x7f43481406b0 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480ad080 con 0x7f43480b6b00 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434f8e6640 10 monclient(hunting): authenticate will time out at 760.790567s 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434de83640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f43481406b0 0x7f43481e1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60504/0 (socket says 172.21.15.174:60504) 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f434de83640 1 -- 172.21.15.174:0/2211392617 learned_addr learned my addr 172.21.15.174:0/2211392617 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 client.?.objecter ms_handle_connect 0x7f43481406b0 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 client.?.objecter ms_handle_connect 0x7f434809ac40 2024-03-29T10:46:05.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 client.?.objecter ms_handle_connect 0x7f43480b6b00 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866299499 0 0) 0x7f4348140b00 con 0x7f43481406b0 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f432c004eb0 con 0x7f43481406b0 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.514+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076156948 0 0) 0x7f43480ad080 con 0x7f43480b6b00 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348140b00 con 0x7f43480b6b00 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413249504 0 0) 0x7f43481d7e20 con 0x7f434809ac40 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43480ad080 con 0x7f434809ac40 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604841836 0 0) 0x7f432c004eb0 con 0x7f43481406b0 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43481d7e20 con 0x7f43481406b0 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4340001af0 con 0x7f43481406b0 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.514 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616078108 0 0) 0x7f4348140b00 con 0x7f43480b6b00 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f432c004eb0 con 0x7f43480b6b00 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4338001b80 con 0x7f43480b6b00 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.516 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379894427 0 0) 0x7f43480ad080 con 0x7f434809ac40 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4348140b00 con 0x7f434809ac40 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4344001930 con 0x7f434809ac40 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.517 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733762319 0 0) 0x7f43481d7e20 con 0x7f43481406b0 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 >> v1:172.21.15.64:6790/0 conn(0x7f43480b6b00 legacy=0x7f434813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 >> v1:172.21.15.64:6789/0 conn(0x7f434809ac40 legacy=0x7f43481d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 monclient: found mon.b 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43481d7ab0 con 0x7f43481406b0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 10 monclient: _renew_subs 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43481e3570 con 0x7f43481406b0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4340001fe0 con 0x7f43481406b0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4340003b30 con 0x7f43481406b0 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 monclient: dump: 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.520 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 10 monclient: _renew_subs 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433f7fe640 4 set_mon_vals no callback set 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43481e38e0 con 0x7f43481406b0 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f434f8e6640 1 librados: init done 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f43400040e0 con 0x7f43481406b0 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1450266794 0 0) 0x7f4340004390 con 0x7f43481406b0 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 10 client.4123.objecter ms_dispatch 0x7f4348000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.515+0000 7f433dffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.516+0000 7f434f8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.516+0000 7f434f8e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.521 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.516+0000 7f434f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.522 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.516+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434809b020 con 0x7f43481406b0 2024-03-29T10:46:05.522 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.518+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f43400045d0 con 0x7f43481406b0 2024-03-29T10:46:05.522 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.518+0000 7f433dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.522 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.518+0000 7f433dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.536 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.552+0000 7f589663e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:46:05.548 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.566+0000 7f0fafa3c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:46:05.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.589+0000 7f434f8e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ab146959-79eb-4128-b51c-c83a6258a4fa", "id": 2}] 2024-03-29T10:46:05.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.589+0000 7f434f8e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab146959-79eb-4128-b51c-c83a6258a4fa", "id": 2}] 2024-03-29T10:46:05.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.589+0000 7f434f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.589+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ab146959-79eb-4128-b51c-c83a6258a4fa", "id": 2} v 0) v1 -- 0x7f434804f730 con 0x7f43481406b0 2024-03-29T10:46:05.579 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.596+0000 7f433dffb640 1 -- 172.21.15.174:0/2211392617 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab146959-79eb-4128-b51c-c83a6258a4fa", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3883977565 0 216599125) 0x7f4340005b40 con 0x7f43481406b0 2024-03-29T10:46:05.579 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.596+0000 7f433dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ab146959-79eb-4128-b51c-c83a6258a4fa", "id": 2}] 2024-03-29T10:46:05.579 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.596+0000 7f433dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:05.579 INFO:teuthology.orchestra.run.smithi174.stdout:2 2024-03-29T10:46:05.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 10 librados: watch_flush enter 2024-03-29T10:46:05.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 10 librados: watch_flush exit 2024-03-29T10:46:05.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 10 monclient: shutdown 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 >> v1:172.21.15.174:6789/0 conn(0x7f43481406b0 legacy=0x7f43481e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 shutdown_connections 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 >> 172.21.15.174:0/2211392617 conn(0x7f4348066b00 msgr2=0x7f43480b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 shutdown_connections 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 1 -- 172.21.15.174:0/2211392617 wait complete. 2024-03-29T10:46:05.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.597+0000 7f434f8e6640 1 librados: shutdown 2024-03-29T10:46:05.590 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 12211654-354e-4112-8f60-e176e2942e06 3 2024-03-29T10:46:05.593 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.610+0000 7f589663e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:46:05.605 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.623+0000 7f0fafa3c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:46:05.639 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.656+0000 7f589663e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:46:05.653 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.670+0000 7f0fafa3c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:46:05.690 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.707+0000 7f7683a3e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.690 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.707+0000 7f7683a3e640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.690 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.707+0000 7f7683a3e640 10 monclient: monmap: 2024-03-29T10:46:05.690 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.690 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.690 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:05.708622+0000 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:05.708622+0000 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.691 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.708+0000 7f7683a3e640 1 Processor -- start 2024-03-29T10:46:05.697 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.712+0000 7f589663e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:46:05.699 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.708+0000 7f7683a3e640 1 -- start start 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.708+0000 7f7683a3e640 10 monclient: init 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.708+0000 7f7683a3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.708+0000 7f7683a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.708+0000 7f7683a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f767c09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f767c09c870 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f767c0b4360 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c09dec0 con 0x7f767c09ac40 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c09da30 con 0x7f767c0b4360 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0b39c0 con 0x7f767c09c870 2024-03-29T10:46:05.700 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 10 monclient(hunting): authenticate will time out at 760.985001s 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7681fdb640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f767c0b4360 0x7f767c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60514/0 (socket says 172.21.15.174:60514) 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7681fdb640 1 -- 172.21.15.174:0/174975058 learned_addr learned my addr 172.21.15.174:0/174975058 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940193145 0 0) 0x7f767c09da30 con 0x7f767c0b4360 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7668003520 con 0x7f767c0b4360 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554262972 0 0) 0x7f767c0b39c0 con 0x7f767c09c870 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f767c09da30 con 0x7f767c09c870 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122148004 0 0) 0x7f7678001110 con 0x7f767c09ac40 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f767c0b39c0 con 0x7f767c09ac40 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3869361442 0 0) 0x7f7668003520 con 0x7f767c0b4360 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 >> v1:172.21.15.64:6790/0 conn(0x7f767c09c870 legacy=0x7f767c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 >> v1:172.21.15.64:6789/0 conn(0x7f767c09ac40 legacy=0x7f767c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 monclient: found mon.noname-c 2024-03-29T10:46:05.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0b5160 con 0x7f767c0b4360 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7674001a10 con 0x7f767c0b4360 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: dump: 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.703 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7674001bf0 con 0x7f767c0b4360 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 1 -- 172.21.15.174:0/174975058 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7674002000 con 0x7f767c0b4360 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7683a3e640 4 set_mon_vals no callback set 2024-03-29T10:46:05.704 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.705 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.705 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.709+0000 7f7673fff640 10 monclient: dump: 2024-03-29T10:46:05.705 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.705 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.706 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 10 monclient: shutdown 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 1 -- 172.21.15.174:0/174975058 >> v1:172.21.15.174:6789/0 conn(0x7f767c0b4360 legacy=0x7f767c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 1 -- 172.21.15.174:0/174975058 shutdown_connections 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 1 -- 172.21.15.174:0/174975058 >> 172.21.15.174:0/174975058 conn(0x7f767c066b00 msgr2=0x7f767c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 1 -- 172.21.15.174:0/174975058 shutdown_connections 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.710+0000 7f7683a3e640 1 -- 172.21.15.174:0/174975058 wait complete. 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 10 monclient: monmap: 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:05.707 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 1 librados: starting msgr at 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 1 librados: starting objecter 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 1 Processor -- start 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 1 -- start start 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 1 librados: setting wanted keys 2024-03-29T10:46:05.708 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 1 librados: calling monclient init 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.711+0000 7f7683a3e640 10 monclient: init 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f767c09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f767c0aee50 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f767c0ae810 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c1d7b20 con 0x7f767c09ac40 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0b00d0 con 0x7f767c0ae810 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0ada70 con 0x7f767c0aee50 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient(hunting): authenticate will time out at 760.988171s 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7681fdb640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f767c0ae810 0x7f767c0b10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60526/0 (socket says 172.21.15.174:60526) 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7681fdb640 1 -- 172.21.15.174:0/3312606686 learned_addr learned my addr 172.21.15.174:0/3312606686 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 client.?.objecter ms_handle_connect 0x7f767c0ae810 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 client.?.objecter ms_handle_connect 0x7f767c0aee50 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383169325 0 0) 0x7f767c0b00d0 con 0x7f767c0ae810 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:46:05.710 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7660004eb0 con 0x7f767c0ae810 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627167180 0 0) 0x7f767c0ada70 con 0x7f767c0aee50 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f767c0b00d0 con 0x7f767c0aee50 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1504408814 0 0) 0x7f7660004eb0 con 0x7f767c0ae810 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f767c0ada70 con 0x7f767c0ae810 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7674001f40 con 0x7f767c0ae810 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596470646 0 0) 0x7f767c0b00d0 con 0x7f767c0aee50 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7660004eb0 con 0x7f767c0aee50 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f766c001930 con 0x7f767c0aee50 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.713 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2880995323 0 0) 0x7f767c0ada70 con 0x7f767c0ae810 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 >> v1:172.21.15.64:6790/0 conn(0x7f767c0aee50 legacy=0x7f767c0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 >> v1:172.21.15.64:6789/0 conn(0x7f767c09ac40 legacy=0x7f767c1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 monclient: found mon.b 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.714 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c1d7640 con 0x7f767c0ae810 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient: _renew_subs 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.712+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c0b17b0 con 0x7f767c0ae810 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7674004e10 con 0x7f767c0ae810 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7674005120 con 0x7f767c0ae810 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 10 monclient: dump: 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.716 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f76737fe640 4 set_mon_vals no callback set 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7683a3e640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7683a3e640 10 monclient: _renew_subs 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7683a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c0b1b50 con 0x7f767c0ae810 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7683a3e640 1 librados: init done 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7674005610 con 0x7f767c0ae810 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3028347236 0 0) 0x7f7674005d50 con 0x7f767c0ae810 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 10 client.4129.objecter ms_dispatch 0x7f767c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:46:05.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.713+0000 7f7671ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.714+0000 7f7683a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.714+0000 7f7683a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.714+0000 7f7683a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.714+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c09b020 con 0x7f767c0ae810 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.717+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f76740058c0 con 0x7f767c0ae810 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.717+0000 7f7671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.717+0000 7f7671ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.719 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.728+0000 7f0fafa3c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:46:05.740 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.757+0000 7f589663e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:46:05.754 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.771+0000 7f0fafa3c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:46:05.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.787+0000 7f7683a3e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "12211654-354e-4112-8f60-e176e2942e06", "id": 3}] 2024-03-29T10:46:05.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.787+0000 7f7683a3e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "12211654-354e-4112-8f60-e176e2942e06", "id": 3}] 2024-03-29T10:46:05.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.787+0000 7f7683a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.770 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.787+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "12211654-354e-4112-8f60-e176e2942e06", "id": 3} v 0) v1 -- 0x7f767c04f730 con 0x7f767c0ae810 2024-03-29T10:46:05.778 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7671ffb640 1 -- 172.21.15.174:0/3312606686 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "12211654-354e-4112-8f60-e176e2942e06", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3509318602 0 525046306) 0x7f7674005f10 con 0x7f767c0ae810 2024-03-29T10:46:05.778 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "12211654-354e-4112-8f60-e176e2942e06", "id": 3}] 2024-03-29T10:46:05.778 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7671ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:05.779 INFO:teuthology.orchestra.run.smithi174.stdout:3 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7683a3e640 10 librados: watch_flush enter 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7683a3e640 10 librados: watch_flush exit 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7683a3e640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.796+0000 7f7683a3e640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 10 monclient: shutdown 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 >> v1:172.21.15.174:6789/0 conn(0x7f767c0ae810 legacy=0x7f767c0b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 shutdown_connections 2024-03-29T10:46:05.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 >> 172.21.15.174:0/3312606686 conn(0x7f767c066b00 msgr2=0x7f767c13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 shutdown_connections 2024-03-29T10:46:05.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 1 -- 172.21.15.174:0/3312606686 wait complete. 2024-03-29T10:46:05.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.797+0000 7f7683a3e640 1 librados: shutdown 2024-03-29T10:46:05.789 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7 4 2024-03-29T10:46:05.861 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.878+0000 7f589663e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:46:05.875 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.892+0000 7f0fafa3c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:46:05.928 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.945+0000 7f81a653b640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.928 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.945+0000 7f81a653b640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.945+0000 7f81a653b640 10 monclient: monmap: 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:05.946782+0000 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:05.946782+0000 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:05.929 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.936 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.951+0000 7f589663e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.946+0000 7f81a653b640 1 Processor -- start 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.946+0000 7f81a653b640 1 -- start start 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.946+0000 7f81a653b640 10 monclient: init 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient(hunting): picked mon.noname-a con 0x7f81a00af360 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient(hunting): picked mon.noname-b con 0x7f81a006af60 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient(hunting): picked mon.noname-c con 0x7f81a0145e60 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00b0520 con 0x7f81a00af360 2024-03-29T10:46:05.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00aff20 con 0x7f81a0145e60 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00b1da0 con 0x7f81a006af60 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 10 monclient(hunting): authenticate will time out at 761.223250s 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a4ad8640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f81a0145e60 0x7f81a014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60536/0 (socket says 172.21.15.174:60536) 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a4ad8640 1 -- 172.21.15.174:0/857486970 learned_addr learned my addr 172.21.15.174:0/857486970 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315917765 0 0) 0x7f81a00aff20 con 0x7f81a0145e60 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8188003520 con 0x7f81a0145e60 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341533799 0 0) 0x7f81a00b0520 con 0x7f81a00af360 2024-03-29T10:46:05.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a00aff20 con 0x7f81a00af360 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925431715 0 0) 0x7f81a00b1da0 con 0x7f81a006af60 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a00b0520 con 0x7f81a006af60 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2359750156 0 0) 0x7f8188003520 con 0x7f81a0145e60 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 >> v1:172.21.15.64:6790/0 conn(0x7f81a006af60 legacy=0x7f81a0145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 >> v1:172.21.15.64:6789/0 conn(0x7f81a00af360 legacy=0x7f81a006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 monclient: found mon.noname-c 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a00b2080 con 0x7f81a0145e60 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f81a653b640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8198001a10 con 0x7f81a0145e60 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.947+0000 7f819effd640 10 monclient: dump: 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8198001bf0 con 0x7f81a0145e60 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f819effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f819effd640 1 -- 172.21.15.174:0/857486970 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8198001f80 con 0x7f81a0145e60 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 4 set_mon_vals no callback set 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f819effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f819effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f819effd640 10 monclient: dump: 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.945 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 10 monclient: shutdown 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 1 -- 172.21.15.174:0/857486970 >> v1:172.21.15.174:6789/0 conn(0x7f81a0145e60 legacy=0x7f81a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 1 -- 172.21.15.174:0/857486970 shutdown_connections 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 1 -- 172.21.15.174:0/857486970 >> 172.21.15.174:0/857486970 conn(0x7f81a00b58f0 msgr2=0x7f81a00b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 1 -- 172.21.15.174:0/857486970 shutdown_connections 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.948+0000 7f81a653b640 1 -- 172.21.15.174:0/857486970 wait complete. 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 10 monclient: monmap: 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:05.946 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 1 librados: starting msgr at 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 1 librados: starting objecter 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 1 Processor -- start 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 1 -- start start 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 1 librados: setting wanted keys 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 1 librados: calling monclient init 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.949+0000 7f81a653b640 10 monclient: init 2024-03-29T10:46:05.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient(hunting): picked mon.noname-a con 0x7f81a006af60 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient(hunting): picked mon.noname-c con 0x7f81a00af360 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient(hunting): picked mon.noname-b con 0x7f81a0145e60 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a0121810 con 0x7f81a006af60 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a01dae60 con 0x7f81a0145e60 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a011f3b0 con 0x7f81a00af360 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a653b640 10 monclient(hunting): authenticate will time out at 761.226327s 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a4ad8640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f81a0145e60 0x7f81a01e27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60550/0 (socket says 172.21.15.174:60550) 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f81a4ad8640 1 -- 172.21.15.174:0/4247887644 learned_addr learned my addr 172.21.15.174:0/4247887644 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 client.?.objecter ms_handle_connect 0x7f81a0145e60 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.948 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 client.?.objecter ms_handle_connect 0x7f81a006af60 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294328087 0 0) 0x7f81a01dae60 con 0x7f81a0145e60 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a0145e60 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 client.?.objecter ms_handle_connect 0x7f81a00af360 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592063415 0 0) 0x7f81a0121810 con 0x7f81a006af60 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:46:05.949 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a01dae60 con 0x7f81a006af60 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967650045 0 0) 0x7f81a011f3b0 con 0x7f81a00af360 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a0121810 con 0x7f81a00af360 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1087091853 0 0) 0x7f818c004eb0 con 0x7f81a0145e60 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a011f3b0 con 0x7f81a0145e60 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8198001f00 con 0x7f81a0145e60 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): dump: 2024-03-29T10:46:05.950 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.950+0000 7f819cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2232776201 0 0) 0x7f81a01dae60 con 0x7f81a006af60 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a006af60 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8194001930 con 0x7f81a006af60 2024-03-29T10:46:05.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient(hunting): dump: 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 856332910 0 0) 0x7f81a0121810 con 0x7f81a00af360 2024-03-29T10:46:05.952 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a01dae60 con 0x7f81a00af360 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8190001830 con 0x7f81a00af360 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient(hunting): dump: 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.953 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3201257208 0 0) 0x7f81a011f3b0 con 0x7f81a0145e60 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 >> v1:172.21.15.64:6790/0 conn(0x7f81a00af360 legacy=0x7f81a01d6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 >> v1:172.21.15.64:6789/0 conn(0x7f81a006af60 legacy=0x7f81a013e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 monclient: found mon.b 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0120810 con 0x7f81a0145e60 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 10 monclient: _renew_subs 2024-03-29T10:46:05.954 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a01da8e0 con 0x7f81a0145e60 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 10 monclient: _renew_subs 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a01dac30 con 0x7f81a0145e60 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f81a653b640 1 librados: init done 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8198001570 con 0x7f81a0145e60 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8198004fb0 con 0x7f81a0145e60 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819e7fc640 4 set_mon_vals no callback set 2024-03-29T10:46:05.955 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 monclient: dump: 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f81980054a0 con 0x7f81a0145e60 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3056800231 0 0) 0x7f8198006b90 con 0x7f81a0145e60 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 10 client.4135.objecter ms_dispatch 0x7f81a0000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:46:05.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.951+0000 7f819cff9640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.952+0000 7f81a653b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.952+0000 7f81a653b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.952+0000 7f81a653b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.952+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8180005080 con 0x7f81a0145e60 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.954+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8198006680 con 0x7f81a0145e60 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.954+0000 7f819cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.957 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:05.954+0000 7f819cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.958 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:05.976+0000 7f0fafa3c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:46:05.982 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:05.999+0000 7f589663e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:46:06.010 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.028+0000 7f0fafa3c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:46:06.026 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.044+0000 7f81a653b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7", "id": 4}] 2024-03-29T10:46:06.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.044+0000 7f81a653b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7", "id": 4}] 2024-03-29T10:46:06.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.044+0000 7f81a653b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.027 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.044+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7", "id": 4} v 0) v1 -- 0x7f81800058b0 con 0x7f81a0145e60 2024-03-29T10:46:06.030 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.048+0000 7f819cff9640 1 -- 172.21.15.174:0/4247887644 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2607828153 0 1697012583) 0x7f81980069f0 con 0x7f81a0145e60 2024-03-29T10:46:06.030 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.048+0000 7f819cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7", "id": 4}] 2024-03-29T10:46:06.030 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.048+0000 7f819cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:06.031 INFO:teuthology.orchestra.run.smithi174.stdout:4 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.048+0000 7f81a653b640 10 librados: watch_flush enter 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 10 librados: watch_flush exit 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 10 monclient: shutdown 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 >> v1:172.21.15.174:6789/0 conn(0x7f81a0145e60 legacy=0x7f81a01e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.032 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 shutdown_connections 2024-03-29T10:46:06.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 >> 172.21.15.174:0/4247887644 conn(0x7f81a00b58f0 msgr2=0x7f81a00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 shutdown_connections 2024-03-29T10:46:06.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 1 -- 172.21.15.174:0/4247887644 wait complete. 2024-03-29T10:46:06.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.049+0000 7f81a653b640 1 librados: shutdown 2024-03-29T10:46:06.037 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.054+0000 7f589663e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:46:06.041 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 42e46429-40bb-49c0-83a3-74d0047b0f4e 5 2024-03-29T10:46:06.066 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.083+0000 7f0fafa3c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:46:06.087 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.103+0000 7f589663e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:46:06.115 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.133+0000 7f0fafa3c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:46:06.140 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.158+0000 7fa161aff640 10 monclient: get_monmap_and_config 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.158+0000 7fa161aff640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.158+0000 7fa161aff640 10 monclient: monmap: 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:06.159303+0000 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:06.159303+0000 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:06.141 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 1 Processor -- start 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 1 -- start start 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient: init 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient(hunting): picked mon.noname-a con 0x7fa15c09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient(hunting): picked mon.noname-c con 0x7fa15c09c870 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient(hunting): picked mon.noname-b con 0x7fa15c0b4360 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c09dec0 con 0x7fa15c09ac40 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c09da30 con 0x7fa15c09c870 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0b39c0 con 0x7fa15c0b4360 2024-03-29T10:46:06.151 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa161aff640 10 monclient(hunting): authenticate will time out at 761.435652s 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa15affd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa15c09c870 0x7fa15c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60558/0 (socket says 172.21.15.174:60558) 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.159+0000 7fa15affd640 1 -- 172.21.15.174:0/4221422873 learned_addr learned my addr 172.21.15.174:0/4221422873 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925969692 0 0) 0x7fa15c09da30 con 0x7fa15c09c870 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa144003520 con 0x7fa15c09c870 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062764023 0 0) 0x7fa15c0b39c0 con 0x7fa15c0b4360 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.152 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c09da30 con 0x7fa15c0b4360 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889710197 0 0) 0x7fa14c001110 con 0x7fa15c09ac40 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c0b39c0 con 0x7fa15c09ac40 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478333333 0 0) 0x7fa144003520 con 0x7fa15c09c870 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 >> v1:172.21.15.64:6790/0 conn(0x7fa15c0b4360 legacy=0x7fa15c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 >> v1:172.21.15.64:6789/0 conn(0x7fa15c09ac40 legacy=0x7fa15c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.153 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0b5160 con 0x7fa15c09c870 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa150001690 con 0x7fa15c09c870 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: dump: 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.154 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa150001870 con 0x7fa15c09c870 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 1 -- 172.21.15.174:0/4221422873 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa150001c80 con 0x7fa15c09c870 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa15a7fc640 10 monclient: dump: 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.155 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 4 set_mon_vals no callback set 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 10 monclient: shutdown 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.160+0000 7fa161aff640 1 -- 172.21.15.174:0/4221422873 >> v1:172.21.15.174:6789/0 conn(0x7fa15c09c870 legacy=0x7fa15c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 -- 172.21.15.174:0/4221422873 shutdown_connections 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 -- 172.21.15.174:0/4221422873 >> 172.21.15.174:0/4221422873 conn(0x7fa15c066b00 msgr2=0x7fa15c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.156 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 -- 172.21.15.174:0/4221422873 shutdown_connections 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 -- 172.21.15.174:0/4221422873 wait complete. 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 10 monclient: monmap: 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.157 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 librados: starting msgr at 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 librados: starting objecter 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.161+0000 7fa161aff640 1 Processor -- start 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 1 -- start start 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 1 librados: setting wanted keys 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 1 librados: calling monclient init 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient: init 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient(hunting): picked mon.noname-c con 0x7fa15c09ac40 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient(hunting): picked mon.noname-a con 0x7fa15c0af2a0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient(hunting): picked mon.noname-b con 0x7fa15c1d7ca0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c1d7100 con 0x7fa15c0af2a0 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0b1900 con 0x7fa15c1d7ca0 2024-03-29T10:46:06.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0acd00 con 0x7fa15c09ac40 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa161aff640 10 monclient(hunting): authenticate will time out at 761.438668s 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa15bfff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa15c1d7ca0 0x7fa15c1390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60568/0 (socket says 172.21.15.174:60568) 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa15affd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa15c0af2a0 0x7fa15c1d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.174:40790/0 (socket says 172.21.15.174:40790) 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa15bfff640 1 -- 172.21.15.174:0/3737841331 learned_addr learned my addr 172.21.15.174:0/3737841331 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa143fff640 10 client.?.objecter ms_handle_connect 0x7fa15c1d7ca0 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa143fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa143fff640 10 client.?.objecter ms_handle_connect 0x7fa15c09ac40 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa143fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa143fff640 10 client.?.objecter ms_handle_connect 0x7fa15c0af2a0 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.162+0000 7fa143fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236512876 0 0) 0x7fa15c0acd00 con 0x7fa15c09ac40 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa148004eb0 con 0x7fa15c09ac40 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446430268 0 0) 0x7fa15c0b1900 con 0x7fa15c1d7ca0 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c0acd00 con 0x7fa15c1d7ca0 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531774473 0 0) 0x7fa15c1d7100 con 0x7fa15c0af2a0 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:46:06.160 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c0b1900 con 0x7fa15c0af2a0 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975728927 0 0) 0x7fa148004eb0 con 0x7fa15c09ac40 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa15c1d7100 con 0x7fa15c09ac40 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa14c0018e0 con 0x7fa15c09ac40 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): dump: 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.161 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2509525072 0 0) 0x7fa15c0b1900 con 0x7fa15c0af2a0 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa148004eb0 con 0x7fa15c0af2a0 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa1500019e0 con 0x7fa15c0af2a0 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): dump: 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348793882 0 0) 0x7fa15c0acd00 con 0x7fa15c1d7ca0 2024-03-29T10:46:06.163 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa15c0b1900 con 0x7fa15c1d7ca0 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa154001af0 con 0x7fa15c1d7ca0 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient(hunting): dump: 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.164 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 922876977 0 0) 0x7fa15c1d7100 con 0x7fa15c09ac40 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 >> v1:172.21.15.174:6789/0 conn(0x7fa15c1d7ca0 legacy=0x7fa15c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 >> v1:172.21.15.64:6789/0 conn(0x7fa15c0af2a0 legacy=0x7fa15c1d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 monclient: found mon.c 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0ace90 con 0x7fa15c09ac40 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 5 monclient: authenticate success, global_id 4139 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 10 monclient: _renew_subs 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:06.165 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c0b0600 con 0x7fa15c09ac40 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 10 monclient: _renew_subs 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c1399a0 con 0x7fa15c09ac40 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa161aff640 1 librados: init done 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa14c001d70 con 0x7fa15c09ac40 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa14c0037b0 con 0x7fa15c09ac40 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa159ffb640 4 set_mon_vals no callback set 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 monclient: dump: 2024-03-29T10:46:06.166 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa14c003ca0 con 0x7fa15c09ac40 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2476446 0 0) 0x7fa14c0054c0 con 0x7fa15c09ac40 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 10 client.4139.objecter ms_dispatch 0x7fa15c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 3 client.4139.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 3 client.4139.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:46:06.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.163+0000 7fa143fff640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.164+0000 7fa161aff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.164+0000 7fa161aff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.164+0000 7fa161aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.164+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c005080 con 0x7fa15c09ac40 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.166+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa14c004fa0 con 0x7fa15c09ac40 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.166+0000 7fa143fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.166+0000 7fa143fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:06.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.238+0000 7fa161aff640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "42e46429-40bb-49c0-83a3-74d0047b0f4e", "id": 5}] 2024-03-29T10:46:06.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.238+0000 7fa161aff640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "42e46429-40bb-49c0-83a3-74d0047b0f4e", "id": 5}] 2024-03-29T10:46:06.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.238+0000 7fa161aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:06.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.238+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd new", "uuid": "42e46429-40bb-49c0-83a3-74d0047b0f4e", "id": 5} v 0) v1 -- 0x7fa13c0058b0 con 0x7fa15c09ac40 2024-03-29T10:46:06.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.241+0000 7fa143fff640 1 -- 172.21.15.174:0/3737841331 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42e46429-40bb-49c0-83a3-74d0047b0f4e", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1249023891 0 1988412176) 0x7fa14c0052b0 con 0x7fa15c09ac40 2024-03-29T10:46:06.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.241+0000 7fa143fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "42e46429-40bb-49c0-83a3-74d0047b0f4e", "id": 5}] 2024-03-29T10:46:06.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.241+0000 7fa143fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:06.224 INFO:teuthology.orchestra.run.smithi174.stdout:5 2024-03-29T10:46:06.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 10 librados: watch_flush enter 2024-03-29T10:46:06.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 10 librados: watch_flush exit 2024-03-29T10:46:06.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 10 client.4139.objecter successfully canceled tick 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 10 monclient: shutdown 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 >> v1:172.21.15.64:6790/0 conn(0x7fa15c09ac40 legacy=0x7fa15c0afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 shutdown_connections 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 >> 172.21.15.174:0/3737841331 conn(0x7fa15c066b00 msgr2=0x7fa15c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 shutdown_connections 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 1 -- 172.21.15.174:0/3737841331 wait complete. 2024-03-29T10:46:06.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.242+0000 7fa161aff640 1 librados: shutdown 2024-03-29T10:46:06.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 03e1d62c-2309-4141-971a-b6f48b5dd70c 6 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.393+0000 7fa5b1e31640 10 monclient: get_monmap_and_config 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.393+0000 7fa5b1e31640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.393+0000 7fa5b1e31640 10 monclient: monmap: 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:06.394467+0000 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:06.394467+0000 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.376 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:06.377 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:06.377 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 1 Processor -- start 2024-03-29T10:46:06.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 1 -- start start 2024-03-29T10:46:06.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 10 monclient: init 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ac09ac40 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ac09c870 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.394+0000 7fa5b1e31640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ac0b4360 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac09dec0 con 0x7fa5ac09ac40 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac09da30 con 0x7fa5ac0b4360 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0b39c0 con 0x7fa5ac09c870 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 10 monclient(hunting): authenticate will time out at 761.670977s 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5abfff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa5ac0b4360 0x7fa5ac14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60574/0 (socket says 172.21.15.174:60574) 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5abfff640 1 -- 172.21.15.174:0/1718484470 learned_addr learned my addr 172.21.15.174:0/1718484470 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322113412 0 0) 0x7fa5ac09da30 con 0x7fa5ac0b4360 2024-03-29T10:46:06.387 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa594003520 con 0x7fa5ac0b4360 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681399114 0 0) 0x7fa5ac0b39c0 con 0x7fa5ac09c870 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac09da30 con 0x7fa5ac09c870 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666559387 0 0) 0x7fa5a0001110 con 0x7fa5ac09ac40 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac0b39c0 con 0x7fa5ac09ac40 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3111805714 0 0) 0x7fa594003520 con 0x7fa5ac0b4360 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 >> v1:172.21.15.64:6790/0 conn(0x7fa5ac09c870 legacy=0x7fa5ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 >> v1:172.21.15.64:6789/0 conn(0x7fa5ac09ac40 legacy=0x7fa5ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 monclient: found mon.noname-c 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0b5160 con 0x7fa5ac0b4360 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa5a4001a10 con 0x7fa5ac0b4360 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: dump: 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.390 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a4001bf0 con 0x7fa5ac0b4360 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 1 -- 172.21.15.174:0/1718484470 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa5a4002000 con 0x7fa5ac0b4360 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5b1e31640 4 set_mon_vals no callback set 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.395+0000 7fa5aa7fc640 10 monclient: dump: 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.391 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 10 monclient: shutdown 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 1 -- 172.21.15.174:0/1718484470 >> v1:172.21.15.174:6789/0 conn(0x7fa5ac0b4360 legacy=0x7fa5ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 1 -- 172.21.15.174:0/1718484470 shutdown_connections 2024-03-29T10:46:06.392 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 1 -- 172.21.15.174:0/1718484470 >> 172.21.15.174:0/1718484470 conn(0x7fa5ac066b00 msgr2=0x7fa5ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 1 -- 172.21.15.174:0/1718484470 shutdown_connections 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.396+0000 7fa5b1e31640 1 -- 172.21.15.174:0/1718484470 wait complete. 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 10 monclient: monmap: 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.393 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 1 librados: starting msgr at 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 1 librados: starting objecter 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 1 Processor -- start 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 1 -- start start 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 1 librados: setting wanted keys 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 1 librados: calling monclient init 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.397+0000 7fa5b1e31640 10 monclient: init 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ac09ac40 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ac0aee50 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ac0ae810 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac1d7b20 con 0x7fa5ac0ae810 2024-03-29T10:46:06.394 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0b00d0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0ada70 con 0x7fa5ac09ac40 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient(hunting): authenticate will time out at 761.674078s 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5aaffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa5ac0aee50 0x7fa5ac0b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60578/0 (socket says 172.21.15.174:60578) 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5aaffd640 1 -- 172.21.15.174:0/135369650 learned_addr learned my addr 172.21.15.174:0/135369650 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 client.?.objecter ms_handle_connect 0x7fa5ac0aee50 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 client.?.objecter ms_handle_connect 0x7fa5ac0ae810 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 client.?.objecter ms_handle_connect 0x7fa5ac09ac40 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561564533 0 0) 0x7fa5ac0b00d0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598004eb0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922694947 0 0) 0x7fa5ac1d7b20 con 0x7fa5ac0ae810 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac0b00d0 con 0x7fa5ac0ae810 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080357172 0 0) 0x7fa5ac0ada70 con 0x7fa5ac09ac40 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.396 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac1d7b20 con 0x7fa5ac09ac40 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1237398541 0 0) 0x7fa598004eb0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ac0ada70 con 0x7fa5ac0aee50 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa59c001960 con 0x7fa5ac0aee50 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): dump: 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.397 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1854741575 0 0) 0x7fa5ac0b00d0 con 0x7fa5ac0ae810 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa598004eb0 con 0x7fa5ac0ae810 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa5a4001f00 con 0x7fa5ac0ae810 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient(hunting): dump: 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.398 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1166982493 0 0) 0x7fa5ac0ada70 con 0x7fa5ac0aee50 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 >> v1:172.21.15.64:6790/0 conn(0x7fa5ac09ac40 legacy=0x7fa5ac1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 >> v1:172.21.15.64:6789/0 conn(0x7fa5ac0ae810 legacy=0x7fa5ac0b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 monclient: found mon.b 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac1d7640 con 0x7fa5ac0aee50 2024-03-29T10:46:06.399 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient: _renew_subs 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.398+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac0b17b0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 10 monclient: _renew_subs 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac0b1b50 con 0x7fa5ac0aee50 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 1 librados: init done 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa59c003610 con 0x7fa5ac0aee50 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.400 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa59c003a20 con 0x7fa5ac0aee50 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5a9ffb640 4 set_mon_vals no callback set 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 10 monclient: dump: 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.401 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa59c003fd0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2158188826 0 0) 0x7fa59c006830 con 0x7fa5ac0aee50 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 10 client.4147.objecter ms_dispatch 0x7fa5ac000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa593fff640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.399+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c005080 con 0x7fa5ac0aee50 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.401+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa59c006a40 con 0x7fa5ac0aee50 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.401+0000 7fa593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.402 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.401+0000 7fa593fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:06.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.473+0000 7fa5b1e31640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "03e1d62c-2309-4141-971a-b6f48b5dd70c", "id": 6}] 2024-03-29T10:46:06.456 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.473+0000 7fa5b1e31640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "03e1d62c-2309-4141-971a-b6f48b5dd70c", "id": 6}] 2024-03-29T10:46:06.456 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.473+0000 7fa5b1e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.456 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.473+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "03e1d62c-2309-4141-971a-b6f48b5dd70c", "id": 6} v 0) v1 -- 0x7fa58c0058b0 con 0x7fa5ac0aee50 2024-03-29T10:46:06.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa593fff640 1 -- 172.21.15.174:0/135369650 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "03e1d62c-2309-4141-971a-b6f48b5dd70c", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3196103017 0 1113811849) 0x7fa59c006180 con 0x7fa5ac0aee50 2024-03-29T10:46:06.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa593fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "03e1d62c-2309-4141-971a-b6f48b5dd70c", "id": 6}] 2024-03-29T10:46:06.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa593fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:06.460 INFO:teuthology.orchestra.run.smithi174.stdout:6 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 10 librados: watch_flush enter 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 10 librados: watch_flush exit 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 10 monclient: shutdown 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.461 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.477+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 >> v1:172.21.15.174:6789/0 conn(0x7fa5ac0aee50 legacy=0x7fa5ac0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.462 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.478+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 shutdown_connections 2024-03-29T10:46:06.462 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.478+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 >> 172.21.15.174:0/135369650 conn(0x7fa5ac066b00 msgr2=0x7fa5ac13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.462 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.478+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 shutdown_connections 2024-03-29T10:46:06.462 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.478+0000 7fa5b1e31640 1 -- 172.21.15.174:0/135369650 wait complete. 2024-03-29T10:46:06.462 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.478+0000 7fa5b1e31640 1 librados: shutdown 2024-03-29T10:46:06.470 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new d61a9d8a-d7d8-4388-9738-50bce2ba2326 7 2024-03-29T10:46:06.488 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.504+0000 7f589663e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:46:06.514 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.532+0000 7f0fafa3c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.587+0000 7fbeae5af640 10 monclient: get_monmap_and_config 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.587+0000 7fbeae5af640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.587+0000 7fbeae5af640 10 monclient: monmap: 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:46:06.588484+0000 2024-03-29T10:46:06.570 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:46:06.588484+0000 2024-03-29T10:46:06.571 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.571 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.571 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.571 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:06.571 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:06.571 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 1 Processor -- start 2024-03-29T10:46:06.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 1 -- start start 2024-03-29T10:46:06.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient: init 2024-03-29T10:46:06.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient(hunting): picked mon.noname-c con 0x7fbea809ac40 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient(hunting): picked mon.noname-a con 0x7fbea809c870 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient(hunting): picked mon.noname-b con 0x7fbea80b4360 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea809dec0 con 0x7fbea809c870 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea809da30 con 0x7fbea809ac40 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80b39c0 con 0x7fbea80b4360 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.588+0000 7fbeae5af640 10 monclient(hunting): authenticate will time out at 761.864900s 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea7fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbea809ac40 0x7fbea80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60590/0 (socket says 172.21.15.174:60590) 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea7fff640 1 -- 172.21.15.174:0/2627927590 learned_addr learned my addr 172.21.15.174:0/2627927590 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399169866 0 0) 0x7fbea809da30 con 0x7fbea809ac40 2024-03-29T10:46:06.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe90003520 con 0x7fbea809ac40 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289269728 0 0) 0x7fbea80b39c0 con 0x7fbea80b4360 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea809da30 con 0x7fbea80b4360 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229322923 0 0) 0x7fbe9c001110 con 0x7fbea809c870 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea80b39c0 con 0x7fbea809c870 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4246159709 0 0) 0x7fbe90003520 con 0x7fbea809ac40 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 >> v1:172.21.15.64:6790/0 conn(0x7fbea80b4360 legacy=0x7fbea814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 >> v1:172.21.15.64:6789/0 conn(0x7fbea809c870 legacy=0x7fbea813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 monclient: found mon.noname-c 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea80b5160 con 0x7fbea809ac40 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.583 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbe98001690 con 0x7fbea809ac40 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbeae5af640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbeae5af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: dump: 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.584 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbeae5af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe98001870 con 0x7fbea809ac40 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 1 -- 172.21.15.174:0/2627927590 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbe98001c80 con 0x7fbea809ac40 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbeae5af640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbeae5af640 4 set_mon_vals no callback set 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.589+0000 7fbea6ffd640 10 monclient: dump: 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.585 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 10 monclient: shutdown 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 1 -- 172.21.15.174:0/2627927590 >> v1:172.21.15.174:6789/0 conn(0x7fbea809ac40 legacy=0x7fbea80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 1 -- 172.21.15.174:0/2627927590 shutdown_connections 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 1 -- 172.21.15.174:0/2627927590 >> 172.21.15.174:0/2627927590 conn(0x7fbea8066b00 msgr2=0x7fbea8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 1 -- 172.21.15.174:0/2627927590 shutdown_connections 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 1 -- 172.21.15.174:0/2627927590 wait complete. 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.590+0000 7fbeae5af640 10 monclient: monmap: 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 0 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 0.000000 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:created 0.000000 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 librados: starting msgr at 2024-03-29T10:46:06.587 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 librados: starting objecter 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 Processor -- start 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 -- start start 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 librados: setting wanted keys 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 librados: calling monclient init 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient: init 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient(hunting): picked mon.noname-c con 0x7fbea809ac40 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient(hunting): picked mon.noname-a con 0x7fbea80acd70 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient(hunting): picked mon.noname-b con 0x7fbea8139820 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80b1a60 con 0x7fbea80acd70 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea81d73f0 con 0x7fbea8139820 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea81d8020 con 0x7fbea809ac40 2024-03-29T10:46:06.588 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.591+0000 7fbeae5af640 10 monclient(hunting): authenticate will time out at 761.867896s 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeacb4c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbea8139820 0x7fbea80b00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:60592/0 (socket says 172.21.15.174:60592) 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeacb4c640 1 -- 172.21.15.174:0/172552928 learned_addr learned my addr 172.21.15.174:0/172552928 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 client.?.objecter ms_handle_connect 0x7fbea8139820 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 client.?.objecter ms_handle_connect 0x7fbea80acd70 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 client.?.objecter ms_handle_connect 0x7fbea809ac40 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821746359 0 0) 0x7fbea81d73f0 con 0x7fbea8139820 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.589 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe94004eb0 con 0x7fbea8139820 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761043320 0 0) 0x7fbea80b1a60 con 0x7fbea80acd70 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea81d73f0 con 0x7fbea80acd70 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170876508 0 0) 0x7fbea81d8020 con 0x7fbea809ac40 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:46:06.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea80b1a60 con 0x7fbea809ac40 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663353343 0 0) 0x7fbe94004eb0 con 0x7fbea8139820 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbea81d8020 con 0x7fbea8139820 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbea0001c00 con 0x7fbea8139820 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.591 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3100570292 0 0) 0x7fbea81d73f0 con 0x7fbea80acd70 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe94004eb0 con 0x7fbea80acd70 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 461441232 0 0) 0x7fbea80b1a60 con 0x7fbea809ac40 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbea81d73f0 con 0x7fbea809ac40 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbe9c001860 con 0x7fbea80acd70 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:06.592 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbe98001b80 con 0x7fbea809ac40 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:06.593 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 219564138 0 0) 0x7fbea81d8020 con 0x7fbea8139820 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 >> v1:172.21.15.64:6790/0 conn(0x7fbea809ac40 legacy=0x7fbea81d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 >> v1:172.21.15.64:6789/0 conn(0x7fbea80acd70 legacy=0x7fbea8139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.594 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 monclient: found mon.b 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea81d81b0 con 0x7fbea8139820 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 10 monclient: _renew_subs 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea80b07b0 con 0x7fbea8139820 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 10 monclient: _renew_subs 2024-03-29T10:46:06.595 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea80b0ce0 con 0x7fbea8139820 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbeae5af640 1 librados: init done 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbea0003990 con 0x7fbea8139820 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbea0003d20 con 0x7fbea8139820 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea67fc640 4 set_mon_vals no callback set 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 10 monclient: dump: 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:epoch 1 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.596 INFO:teuthology.orchestra.run.smithi174.stderr:election_strategy: 1 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.592+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbea00042d0 con 0x7fbea8139820 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.593+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1962404564 0 0) 0x7fbea0006b90 con 0x7fbea8139820 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.593+0000 7fbea4ff9640 10 client.4153.objecter ms_dispatch 0x7fbea8000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.593+0000 7fbea4ff9640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.593+0000 7fbea4ff9640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.593+0000 7fbea4ff9640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.594+0000 7fbeae5af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.594+0000 7fbeae5af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.594+0000 7fbeae5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.597 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.594+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe88005080 con 0x7fbea8139820 2024-03-29T10:46:06.598 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.595+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbea0004710 con 0x7fbea8139820 2024-03-29T10:46:06.598 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.596+0000 7fbea4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.598 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.596+0000 7fbea4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:06.629 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.646+0000 7f589663e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:46:06.649 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.666+0000 7fbeae5af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d61a9d8a-d7d8-4388-9738-50bce2ba2326", "id": 7}] 2024-03-29T10:46:06.649 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.666+0000 7fbeae5af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d61a9d8a-d7d8-4388-9738-50bce2ba2326", "id": 7}] 2024-03-29T10:46:06.649 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.666+0000 7fbeae5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:06.649 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.666+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d61a9d8a-d7d8-4388-9738-50bce2ba2326", "id": 7} v 0) v1 -- 0x7fbe880058b0 con 0x7fbea8139820 2024-03-29T10:46:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.670+0000 7fbea4ff9640 1 -- 172.21.15.174:0/172552928 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d61a9d8a-d7d8-4388-9738-50bce2ba2326", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 470224024 0 1371665406) 0x7fbea0006530 con 0x7fbea8139820 2024-03-29T10:46:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.670+0000 7fbea4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d61a9d8a-d7d8-4388-9738-50bce2ba2326", "id": 7}] 2024-03-29T10:46:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.670+0000 7fbea4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:06.653 INFO:teuthology.orchestra.run.smithi174.stdout:7 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 10 librados: watch_flush enter 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 10 librados: watch_flush exit 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 10 monclient: shutdown 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 >> v1:172.21.15.174:6789/0 conn(0x7fbea8139820 legacy=0x7fbea80b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 shutdown_connections 2024-03-29T10:46:06.654 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 >> 172.21.15.174:0/172552928 conn(0x7fbea8066b00 msgr2=0x7fbea80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.655 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 shutdown_connections 2024-03-29T10:46:06.655 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 1 -- 172.21.15.174:0/172552928 wait complete. 2024-03-29T10:46:06.655 INFO:teuthology.orchestra.run.smithi174.stderr:2024-03-29T10:46:06.671+0000 7fbeae5af640 1 librados: shutdown 2024-03-29T10:46:06.655 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.671+0000 7f0fafa3c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:46:06.664 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:46:06.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:46:06.666 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:46:06.666 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:46:06.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:46:06.668 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:46:06.668 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:46:06.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:46:06.670 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:46:06.670 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:46:06.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:46:06.672 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:46:06.673 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:46:06.673 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:46:06.675 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:46:06.675 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:46:06.675 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:46:06.677 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:46:06.677 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:46:06.677 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:46:06.679 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:46:06.679 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:46:06.680 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:46:06.682 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:46:06.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:06.788 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.805+0000 7f589663e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:46:06.797 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.815+0000 7f0fafa3c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:46:06.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.825+0000 7f6142e81640 10 monclient: get_monmap_and_config 2024-03-29T10:46:06.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.825+0000 7f6142e81640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.825+0000 7f6142e81640 10 monclient: monmap: 2024-03-29T10:46:06.808 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T10:46:06.808 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:46:06.827154+0000 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:46:06.827154+0000 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:06.809 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 1 Processor -- start 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 1 -- start start 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient: init 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient(hunting): picked mon.noname-a con 0x7f613c0b8240 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient(hunting): picked mon.noname-b con 0x7f613c0b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient(hunting): picked mon.noname-c con 0x7f613c09c8c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c09bd80 con 0x7f613c0b8240 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c09b960 con 0x7f613c09c8c0 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c0b2ec0 con 0x7f613c0b1f60 2024-03-29T10:46:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f6142e81640 10 monclient(hunting): authenticate will time out at 778.126776s 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f613bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f613c0b1f60 0x7f613c09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35742/0 (socket says 172.21.15.64:35742) 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.826+0000 7f613bfff640 1 -- 172.21.15.64:0/1128407722 learned_addr learned my addr 172.21.15.64:0/1128407722 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692974950 0 0) 0x7f613c0b2ec0 con 0x7f613c0b1f60 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): my global_id is 4175 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612c003520 con 0x7f613c0b1f60 2024-03-29T10:46:06.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617869961 0 0) 0x7f613c09bd80 con 0x7f613c0b8240 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): my global_id is 4182 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c0b2ec0 con 0x7f613c0b8240 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239905453 0 0) 0x7f613c09b960 con 0x7f613c09c8c0 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c09bd80 con 0x7f613c09c8c0 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 858990079 0 0) 0x7f612c003520 con 0x7f613c0b1f60 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 >> v1:172.21.15.174:6789/0 conn(0x7f613c09c8c0 legacy=0x7f613c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 >> v1:172.21.15.64:6789/0 conn(0x7f613c0b8240 legacy=0x7f613c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c13fb60 con 0x7f613c0b1f60 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6128001690 con 0x7f613c0b1f60 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: dump: 2024-03-29T10:46:06.824 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f6142e81640 5 monclient: authenticate success, global_id 4175 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f6142e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61280017f0 con 0x7f613c0b1f60 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 1 -- 172.21.15.64:0/1128407722 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6128001c40 con 0x7f613c0b1f60 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f6142e81640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f6142e81640 4 set_mon_vals no callback set 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.827+0000 7f613b7fe640 10 monclient: dump: 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.826 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 10 monclient: shutdown 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 1 -- 172.21.15.64:0/1128407722 >> v1:172.21.15.64:6790/0 conn(0x7f613c0b1f60 legacy=0x7f613c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 1 -- 172.21.15.64:0/1128407722 shutdown_connections 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 1 -- 172.21.15.64:0/1128407722 >> 172.21.15.64:0/1128407722 conn(0x7f613c066b00 msgr2=0x7f613c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 1 -- 172.21.15.64:0/1128407722 shutdown_connections 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.828+0000 7f6142e81640 1 -- 172.21.15.64:0/1128407722 wait complete. 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 10 monclient: monmap: 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:06.827 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 1 librados: starting msgr at 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 1 librados: starting objecter 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 1 Processor -- start 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 1 -- start start 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 1 librados: setting wanted keys 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 1 librados: calling monclient init 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.829+0000 7f6142e81640 10 monclient: init 2024-03-29T10:46:06.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient(hunting): picked mon.noname-a con 0x7f613c09c8c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient(hunting): picked mon.noname-b con 0x7f613c0b8240 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient(hunting): picked mon.noname-c con 0x7f613c130e00 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c133200 con 0x7f613c09c8c0 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c133470 con 0x7f613c0b8240 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c1459a0 con 0x7f613c130e00 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6142e81640 10 monclient(hunting): authenticate will time out at 778.130082s 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6140c1d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f613c09c8c0 0x7f613c133890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57650/0 (socket says 172.21.15.64:57650) 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f6140c1d640 1 -- 172.21.15.64:0/80562732 learned_addr learned my addr 172.21.15.64:0/80562732 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 client.?.objecter ms_handle_connect 0x7f613c09c8c0 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 client.?.objecter ms_handle_connect 0x7f613c0b8240 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 client.?.objecter ms_handle_connect 0x7f613c130e00 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927004752 0 0) 0x7f613c133200 con 0x7f613c09c8c0 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6120004eb0 con 0x7f613c09c8c0 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561399233 0 0) 0x7f613c133470 con 0x7f613c0b8240 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c133200 con 0x7f613c0b8240 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067371797 0 0) 0x7f613c1459a0 con 0x7f613c130e00 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): my global_id is 4178 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c133470 con 0x7f613c130e00 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 88361333 0 0) 0x7f6120004eb0 con 0x7f613c09c8c0 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f613c1459a0 con 0x7f613c09c8c0 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6130001990 con 0x7f613c09c8c0 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): dump: 2024-03-29T10:46:06.831 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.832 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3537673755 0 0) 0x7f613c133200 con 0x7f613c0b8240 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6120004eb0 con 0x7f613c0b8240 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907259236 0 0) 0x7f613c133470 con 0x7f613c130e00 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f613c133200 con 0x7f613c130e00 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6128001b00 con 0x7f613c0b8240 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): dump: 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.833 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6134001bf0 con 0x7f613c130e00 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.830+0000 7f61397fa640 10 monclient(hunting): dump: 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.834 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 338529295 0 0) 0x7f613c1459a0 con 0x7f613c09c8c0 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 >> v1:172.21.15.64:6790/0 conn(0x7f613c130e00 legacy=0x7f613c144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 >> v1:172.21.15.174:6789/0 conn(0x7f613c0b8240 legacy=0x7f613c131aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 monclient: found mon.a 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c145b30 con 0x7f613c09c8c0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 5 monclient: authenticate success, global_id 4185 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 10 monclient: _renew_subs 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c1e3120 con 0x7f613c09c8c0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61300035f0 con 0x7f613c09c8c0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6130003900 con 0x7f613c09c8c0 2024-03-29T10:46:06.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 monclient: dump: 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 10 monclient: _renew_subs 2024-03-29T10:46:06.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c1e3490 con 0x7f613c09c8c0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f6142e81640 1 librados: init done 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f613affd640 4 set_mon_vals no callback set 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6130003df0 con 0x7f613c09c8c0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f61300040a0 con 0x7f613c09c8c0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 10 client.4185.objecter ms_dispatch 0x7f613c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.831+0000 7f61397fa640 20 client.4185.objecter dump_active .. 0 homeless 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.832+0000 7f6142e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.832+0000 7f6142e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.832+0000 7f6142e81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.832+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613c09b170 con 0x7f613c09c8c0 2024-03-29T10:46:06.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.834+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f61300043f0 con 0x7f613c09c8c0 2024-03-29T10:46:06.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.834+0000 7f61397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.834+0000 7f61397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:06.839 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.853+0000 7f589663e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:46:06.843 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.861+0000 7f0fafa3c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:46:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.895+0000 7f6142e81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.895+0000 7f6142e81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.895+0000 7f6142e81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.895+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f613c04f730 con 0x7f613c09c8c0 2024-03-29T10:46:06.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.896+0000 7f61397fa640 1 -- 172.21.15.64:0/80562732 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f6130005870 con 0x7f613c09c8c0 2024-03-29T10:46:06.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.896+0000 7f61397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:06.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.896+0000 7f61397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:06.880 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-29T10:46:06.880 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 10 librados: watch_flush enter 2024-03-29T10:46:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 10 librados: watch_flush exit 2024-03-29T10:46:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 20 client.4185.objecter shutdown clearing up homeless session... 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 10 client.4185.objecter successfully canceled tick 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 10 monclient: shutdown 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 >> v1:172.21.15.64:6789/0 conn(0x7f613c09c8c0 legacy=0x7f613c133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 shutdown_connections 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 >> 172.21.15.64:0/80562732 conn(0x7f613c066b00 msgr2=0x7f613c069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 shutdown_connections 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 1 -- 172.21.15.64:0/80562732 wait complete. 2024-03-29T10:46:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-29T10:46:06.897+0000 7f6142e81640 1 librados: shutdown 2024-03-29T10:46:06.890 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:46:06.891 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:46:06.954 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:06.972+0000 7f0fafa3c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:46:06.955 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:06.972+0000 7f589663e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:46:07.050 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:07.067+0000 7f0fafa3c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:46:07.055 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:07.071+0000 7f589663e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:46:07.201 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:07.219+0000 7f0fafa3c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:46:07.216 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:07.232+0000 7f589663e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:46:07.243 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:07.261+0000 7f0fafa3c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:46:07.260 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:07.277+0000 7f589663e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:46:07.274 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-29T10:46:07.291+0000 7f0968017640 -1 Falling back to public interface 2024-03-29T10:46:07.281 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-03-29T10:46:07.298+0000 7fa392615640 -1 Falling back to public interface 2024-03-29T10:46:07.286 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-29T10:46:07.303+0000 7ff23f012640 -1 Falling back to public interface 2024-03-29T10:46:07.288 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-03-29T10:46:07.306+0000 7f229c014640 -1 Falling back to public interface 2024-03-29T10:46:07.289 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-29T10:46:07.306+0000 7f9af22c2640 -1 Falling back to public interface 2024-03-29T10:46:07.289 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-29T10:46:07.306+0000 7f9af1271640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T10:46:07.294 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-03-29T10:46:07.312+0000 7fab71b9a640 -1 Falling back to public interface 2024-03-29T10:46:07.299 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-29T10:46:07.317+0000 7f2b8ea0d640 -1 Falling back to public interface 2024-03-29T10:46:07.305 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-29T10:46:07.322+0000 7fcda5773640 -1 Falling back to public interface 2024-03-29T10:46:07.398 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:07.415+0000 7f0fafa3c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:46:07.422 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:07.440+0000 7f589663e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:46:07.697 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.803+0000 7ff60af39640 10 monclient: get_monmap_and_config 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.803+0000 7ff60af39640 10 monclient: build_initial_monmap 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.803+0000 7ff60af39640 10 monclient: monmap: 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:07.804596+0000 2024-03-29T10:46:07.786 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:07.804596+0000 2024-03-29T10:46:07.787 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:07.787 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.787 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:07.787 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:07.787 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:07.787 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.804+0000 7ff60af39640 1 Processor -- start 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.804+0000 7ff60af39640 1 -- start start 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.804+0000 7ff60af39640 10 monclient: init 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient(hunting): picked mon.noname-a con 0x7ff6041457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient(hunting): picked mon.noname-c con 0x7ff60413f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient(hunting): picked mon.noname-b con 0x7ff6040a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6040aaef0 con 0x7ff6041457f0 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604145bd0 con 0x7ff60413f650 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604145eb0 con 0x7ff6040a9d70 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:07.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient(hunting): authenticate will time out at 779.103955s 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff608cd5640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff6041457f0 0x7ff60413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57654/0 (socket says 172.21.15.64:57654) 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff608cd5640 1 -- 172.21.15.64:0/2996479986 learned_addr learned my addr 172.21.15.64:0/2996479986 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425036505 0 0) 0x7ff6040aaef0 con 0x7ff6041457f0 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f4003520 con 0x7ff6041457f0 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291175152 0 0) 0x7ff604145eb0 con 0x7ff6040a9d70 2024-03-29T10:46:07.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6040aaef0 con 0x7ff6040a9d70 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789678286 0 0) 0x7ff604145bd0 con 0x7ff60413f650 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604145eb0 con 0x7ff60413f650 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833451916 0 0) 0x7ff5f4003520 con 0x7ff6041457f0 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 >> v1:172.21.15.64:6790/0 conn(0x7ff6040a9d70 legacy=0x7ff60414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 >> v1:172.21.15.174:6789/0 conn(0x7ff60413f650 legacy=0x7ff6040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 monclient: found mon.noname-a 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff604146340 con 0x7ff6041457f0 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff5f8001690 con 0x7ff6041457f0 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: dump: 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.802 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5f80017f0 con 0x7ff6041457f0 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 1 -- 172.21.15.64:0/2996479986 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff5f8001c00 con 0x7ff6041457f0 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff60af39640 4 set_mon_vals no callback set 2024-03-29T10:46:07.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.805+0000 7ff6037fe640 10 monclient: dump: 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 10 monclient: shutdown 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:07.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 1 -- 172.21.15.64:0/2996479986 >> v1:172.21.15.64:6789/0 conn(0x7ff6041457f0 legacy=0x7ff60413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 1 -- 172.21.15.64:0/2996479986 shutdown_connections 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 1 -- 172.21.15.64:0/2996479986 >> 172.21.15.64:0/2996479986 conn(0x7ff604066a00 msgr2=0x7ff604066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 1 -- 172.21.15.64:0/2996479986 shutdown_connections 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.806+0000 7ff60af39640 1 -- 172.21.15.64:0/2996479986 wait complete. 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 10 monclient: build_initial_monmap 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 10 monclient: monmap: 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:07.805 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 1 librados: starting msgr at 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 1 librados: starting objecter 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 1 Processor -- start 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 1 -- start start 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 1 librados: setting wanted keys 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 1 librados: calling monclient init 2024-03-29T10:46:07.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.807+0000 7ff60af39640 10 monclient: init 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient(hunting): picked mon.noname-b con 0x7ff6040a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient(hunting): picked mon.noname-c con 0x7ff60413f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient(hunting): picked mon.noname-a con 0x7ff6041457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604140860 con 0x7ff6041457f0 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604140d80 con 0x7ff6040a9d70 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60413fda0 con 0x7ff60413f650 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff60af39640 10 monclient(hunting): authenticate will time out at 779.107282s 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff603fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff60413f650 0x7ff604134ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35776/0 (socket says 172.21.15.64:35776) 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff603fff640 1 -- 172.21.15.64:0/3782337201 learned_addr learned my addr 172.21.15.64:0/3782337201 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 client.?.objecter ms_handle_connect 0x7ff60413f650 2024-03-29T10:46:07.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:07.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 client.?.objecter ms_handle_connect 0x7ff6041457f0 2024-03-29T10:46:07.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:07.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 client.?.objecter ms_handle_connect 0x7ff6040a9d70 2024-03-29T10:46:07.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:07.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804646806 0 0) 0x7ff60413fda0 con 0x7ff60413f650 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e8004eb0 con 0x7ff60413f650 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156979049 0 0) 0x7ff604140860 con 0x7ff6041457f0 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60413fda0 con 0x7ff6041457f0 2024-03-29T10:46:07.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668407852 0 0) 0x7ff604140d80 con 0x7ff6040a9d70 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604140860 con 0x7ff6040a9d70 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.808+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636404083 0 0) 0x7ff5e8004eb0 con 0x7ff60413f650 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff604140d80 con 0x7ff60413f650 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff5f0001a10 con 0x7ff60413f650 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): dump: 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:07.810 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321339015 0 0) 0x7ff60413fda0 con 0x7ff6041457f0 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5e8004eb0 con 0x7ff6041457f0 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff5fc001c30 con 0x7ff6041457f0 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:07.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): dump: 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2405863343 0 0) 0x7ff604140860 con 0x7ff6040a9d70 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff60413fda0 con 0x7ff6040a9d70 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff5f8001b00 con 0x7ff6040a9d70 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient(hunting): dump: 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:07.813 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 493704621 0 0) 0x7ff604140d80 con 0x7ff60413f650 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 >> v1:172.21.15.174:6789/0 conn(0x7ff6040a9d70 legacy=0x7ff6041422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 >> v1:172.21.15.64:6789/0 conn(0x7ff6041457f0 legacy=0x7ff6041e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: _finish_hunting 0 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 monclient: found mon.c 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff604144880 con 0x7ff60413f650 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: _finish_auth 0 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: _check_auth_tickets 2024-03-29T10:46:07.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 5 monclient: authenticate success, global_id 4184 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 10 monclient: _renew_subs 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff604144af0 con 0x7ff60413f650 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 10 monclient: _renew_subs 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff604142bc0 con 0x7ff60413f650 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff60af39640 1 librados: init done 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5f0001cf0 con 0x7ff60413f650 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:07.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff5f0003a60 con 0x7ff60413f650 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff602ffd640 4 set_mon_vals no callback set 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 monclient: dump: 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:07.816 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff5f0004010 con 0x7ff60413f650 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7ff5f0006a20 con 0x7ff60413f650 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 10 client.4184.objecter ms_dispatch 0x7ff604000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.809+0000 7ff6017fa640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.810+0000 7ff60af39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.810+0000 7ff60af39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.810+0000 7ff60af39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:07.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.810+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4005080 con 0x7ff60413f650 2024-03-29T10:46:07.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.812+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff5f0006be0 con 0x7ff60413f650 2024-03-29T10:46:07.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.812+0000 7ff6017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:07.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.812+0000 7ff6017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:07.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff60af39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff60af39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff60af39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:07.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5e4005330 con 0x7ff60413f650 2024-03-29T10:46:07.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff6017fa640 1 -- 172.21.15.64:0/3782337201 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7ff5f0006310 con 0x7ff60413f650 2024-03-29T10:46:07.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff6017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.875+0000 7ff6017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:07.859 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:07.859 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:07.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 10 librados: watch_flush enter 2024-03-29T10:46:07.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 10 librados: watch_flush exit 2024-03-29T10:46:07.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T10:46:07.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 10 client.4184.objecter successfully canceled tick 2024-03-29T10:46:07.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 10 monclient: shutdown 2024-03-29T10:46:07.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:07.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 >> v1:172.21.15.64:6790/0 conn(0x7ff60413f650 legacy=0x7ff604134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 shutdown_connections 2024-03-29T10:46:07.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 >> 172.21.15.64:0/3782337201 conn(0x7ff604066a00 msgr2=0x7ff6040b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:07.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 shutdown_connections 2024-03-29T10:46:07.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 1 -- 172.21.15.64:0/3782337201 wait complete. 2024-03-29T10:46:07.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:07.877+0000 7ff60af39640 1 librados: shutdown 2024-03-29T10:46:07.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:08.194 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:08.212+0000 7f0fafa3c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:46:08.306 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:08.324+0000 7f589663e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:46:08.466 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:08.484+0000 7f0fafa3c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:46:08.516 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:08.534+0000 7f0fafa3c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:46:08.593 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:08.611+0000 7f589663e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:46:08.630 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:08.647+0000 7f0fafa3c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:46:08.648 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:08.666+0000 7f589663e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:46:08.771 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-29T10:46:08.788+0000 7f589663e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:46:09.645 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:09.662+0000 7f0f50711640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:46:09.645 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-03-29T10:46:09.662+0000 7f0f50711640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:46:10.735 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-03-29T10:46:10.752+0000 7f0968017640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:46:10.805 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-03-29T10:46:10.823+0000 7fab71b9a640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:46:10.805 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-03-29T10:46:10.823+0000 7fa392615640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:46:10.813 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-29T10:46:10.830+0000 7f9af22c2640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:46:10.814 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-03-29T10:46:10.832+0000 7f2b8ea0d640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:46:10.817 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-03-29T10:46:10.835+0000 7f229c014640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:46:10.826 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-03-29T10:46:10.843+0000 7fcda5773640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:46:10.829 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-03-29T10:46:10.847+0000 7ff23f012640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:46:14.681 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.854+0000 7f6054a63640 10 monclient: get_monmap_and_config 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.854+0000 7f6054a63640 10 monclient: build_initial_monmap 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.854+0000 7f6054a63640 10 monclient: monmap: 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:14.855206+0000 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:14.855206+0000 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.837 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:14.838 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:14.838 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:14.838 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.855+0000 7f6054a63640 1 Processor -- start 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.855+0000 7f6054a63640 1 -- start start 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.855+0000 7f6054a63640 10 monclient: init 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient(hunting): picked mon.noname-a con 0x7f605006d260 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient(hunting): picked mon.noname-c con 0x7f60500723b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient(hunting): picked mon.noname-b con 0x7f6050146b80 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60500701d0 con 0x7f605006d260 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6050070440 con 0x7f60500723b0 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605006c2c0 con 0x7f6050146b80 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f6054a63640 10 monclient(hunting): authenticate will time out at 786.155513s 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f604ed9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6050146b80 0x7f6050151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:43174/0 (socket says 172.21.15.64:43174) 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.856+0000 7f604ed9d640 1 -- 172.21.15.64:0/212124255 learned_addr learned my addr 172.21.15.64:0/212124255 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168553170 0 0) 0x7f6050070440 con 0x7f60500723b0 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6038003520 con 0x7f60500723b0 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128642107 0 0) 0x7f60500701d0 con 0x7f605006d260 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): my global_id is 4227 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6050070440 con 0x7f605006d260 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94239315 0 0) 0x7f605006c2c0 con 0x7f6050146b80 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 10 monclient(hunting): my global_id is 4223 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.857+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60500701d0 con 0x7f6050146b80 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176808370 0 0) 0x7f6038003520 con 0x7f60500723b0 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 >> v1:172.21.15.64:6790/0 conn(0x7f6050146b80 legacy=0x7f6050151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 >> v1:172.21.15.64:6789/0 conn(0x7f605006d260 legacy=0x7f6050071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 monclient: found mon.noname-c 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605006c7e0 con 0x7f60500723b0 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: _finish_auth 0 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:46:14.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f6054a63640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f6054a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f60440015c0 con 0x7f60500723b0 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: dump: 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.844 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f6054a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60440017e0 con 0x7f60500723b0 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f6054a63640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f6054a63640 4 set_mon_vals no callback set 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 1 -- 172.21.15.64:0/212124255 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6044001b70 con 0x7f60500723b0 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.858+0000 7f604d59a640 10 monclient: dump: 2024-03-29T10:46:14.845 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 10 monclient: shutdown 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:14.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 1 -- 172.21.15.64:0/212124255 >> v1:172.21.15.174:6789/0 conn(0x7f60500723b0 legacy=0x7f6050146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 1 -- 172.21.15.64:0/212124255 shutdown_connections 2024-03-29T10:46:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 1 -- 172.21.15.64:0/212124255 >> 172.21.15.64:0/212124255 conn(0x7f6050067500 msgr2=0x7f60500b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 1 -- 172.21.15.64:0/212124255 shutdown_connections 2024-03-29T10:46:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.859+0000 7f6054a63640 1 -- 172.21.15.64:0/212124255 wait complete. 2024-03-29T10:46:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.860+0000 7f6054a63640 10 monclient: build_initial_monmap 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.860+0000 7f6054a63640 10 monclient: monmap: 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 1 librados: starting msgr at 2024-03-29T10:46:14.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 1 librados: starting objecter 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 1 Processor -- start 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 1 -- start start 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 1 librados: setting wanted keys 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 1 librados: calling monclient init 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.861+0000 7f6054a63640 10 monclient: init 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient(hunting): picked mon.noname-c con 0x7f605006d260 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient(hunting): picked mon.noname-b con 0x7f60500723b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient(hunting): picked mon.noname-a con 0x7f6050146b80 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60501df850 con 0x7f6050146b80 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60500ac2f0 con 0x7f60500723b0 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60501dfaf0 con 0x7f605006d260 2024-03-29T10:46:14.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f6054a63640 10 monclient(hunting): authenticate will time out at 786.161043s 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f604e59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f605006d260 0x7f60501df140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:43286/0 (socket says 172.21.15.64:43286) 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f604e59c640 1 -- 172.21.15.64:0/476284394 learned_addr learned my addr 172.21.15.64:0/476284394 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f603effd640 10 client.?.objecter ms_handle_connect 0x7f605006d260 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.862+0000 7f603effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 client.?.objecter ms_handle_connect 0x7f6050146b80 2024-03-29T10:46:14.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018357206 0 0) 0x7f60501df850 con 0x7f6050146b80 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): my global_id is 4230 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030004eb0 con 0x7f6050146b80 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848598281 0 0) 0x7f60501dfaf0 con 0x7f605006d260 2024-03-29T10:46:14.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): my global_id is 4226 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60501df850 con 0x7f605006d260 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 client.?.objecter ms_handle_connect 0x7f60500723b0 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1780291104 0 0) 0x7f6030004eb0 con 0x7f6050146b80 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60501dfaf0 con 0x7f6050146b80 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6048001c70 con 0x7f6050146b80 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:14.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): dump: 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.863+0000 7f603effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190430844 0 0) 0x7f60500ac2f0 con 0x7f60500723b0 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030004eb0 con 0x7f60500723b0 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638894902 0 0) 0x7f6030004eb0 con 0x7f60500723b0 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60500ac2f0 con 0x7f60500723b0 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6044001500 con 0x7f60500723b0 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient(hunting): dump: 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.857 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3541204598 0 0) 0x7f60501df850 con 0x7f605006d260 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6030004eb0 con 0x7f605006d260 2024-03-29T10:46:14.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 943993146 0 0) 0x7f60501dfaf0 con 0x7f6050146b80 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 >> v1:172.21.15.64:6790/0 conn(0x7f605006d260 legacy=0x7f60501df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 >> v1:172.21.15.174:6789/0 conn(0x7f60500723b0 legacy=0x7f60500addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 monclient: found mon.a 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60500aea40 con 0x7f6050146b80 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient: _finish_auth 0 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f603effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f6054a63640 5 monclient: authenticate success, global_id 4230 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f6054a63640 10 monclient: _renew_subs 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f6054a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.864+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60500aec90 con 0x7f6050146b80 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.865+0000 7f6054a63640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.865+0000 7f6054a63640 10 monclient: _renew_subs 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.865+0000 7f6054a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.865+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60500ae6b0 con 0x7f6050146b80 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.865+0000 7f6054a63640 1 librados: init done 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.865+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6048004b60 con 0x7f6050146b80 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.866+0000 7f603effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.866+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6048004e70 con 0x7f6050146b80 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.866+0000 7f603effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.866+0000 7f603effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.866+0000 7f603effd640 10 monclient: dump: 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:14.860 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.866+0000 7f604cd99640 4 set_mon_vals no callback set 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.867+0000 7f6054a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.867+0000 7f6054a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.867+0000 7f6054a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.867+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6034005080 con 0x7f6050146b80 2024-03-29T10:46:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.869+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f604801ae40 con 0x7f6050146b80 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.870+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f6048084d30 con 0x7f6050146b80 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.870+0000 7f603effd640 10 client.4230.objecter ms_dispatch 0x7f6050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.870+0000 7f603effd640 3 client.4230.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.870+0000 7f603effd640 3 client.4230.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.870+0000 7f603effd640 20 client.4230.objecter dump_active .. 0 homeless 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.870+0000 7f604e59c640 10 monclient: get_auth_request con 0x7f603006e2b0 auth_method 0 2024-03-29T10:46:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.871+0000 7f603effd640 10 client.4230.objecter ms_handle_connect 0x7f603006e2b0 2024-03-29T10:46:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.882+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6048050440 con 0x7f6050146b80 2024-03-29T10:46:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.882+0000 7f603effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:14.882+0000 7f603effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:15.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.284+0000 7f6054a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:15.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.284+0000 7f6054a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:15.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.284+0000 7f6054a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:15.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.284+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6034005330 con 0x7f6050146b80 2024-03-29T10:46:15.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.285+0000 7f603effd640 1 -- 172.21.15.64:0/476284394 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f6048052540 con 0x7f6050146b80 2024-03-29T10:46:15.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.285+0000 7f603effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:15.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.285+0000 7f603effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:15.268 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:15.268 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 10 librados: watch_flush enter 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 10 librados: watch_flush exit 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 20 client.4230.objecter shutdown clearing up homeless session... 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 10 client.4230.objecter successfully canceled tick 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 >> v1:172.21.15.174:6816/2473791532 conn(0x7f603006e2b0 legacy=0x7f6030070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 10 monclient: shutdown 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 >> v1:172.21.15.64:6789/0 conn(0x7f6050146b80 legacy=0x7f60500aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.289+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 shutdown_connections 2024-03-29T10:46:15.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.290+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 >> 172.21.15.64:0/476284394 conn(0x7f6050067500 msgr2=0x7f60500707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:15.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.290+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 shutdown_connections 2024-03-29T10:46:15.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.290+0000 7f6054a63640 1 -- 172.21.15.64:0/476284394 wait complete. 2024-03-29T10:46:15.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:15.290+0000 7f6054a63640 1 librados: shutdown 2024-03-29T10:46:15.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:22.095 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:22.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.255+0000 7f93b7196640 10 monclient: get_monmap_and_config 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.255+0000 7f93b7196640 10 monclient: build_initial_monmap 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.255+0000 7f93b7196640 10 monclient: monmap: 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:22.255962+0000 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:22.255962+0000 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:22.238 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.256+0000 7f93b7196640 1 Processor -- start 2024-03-29T10:46:22.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.256+0000 7f93b7196640 1 -- start start 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.256+0000 7f93b7196640 10 monclient: init 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient(hunting): picked mon.noname-b con 0x7f93b00b0770 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient(hunting): picked mon.noname-a con 0x7f93b014a700 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient(hunting): picked mon.noname-c con 0x7f93b00ac5a0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00aba70 con 0x7f93b014a700 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00aecb0 con 0x7f93b00ac5a0 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b00af150 con 0x7f93b00b0770 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93b7196640 10 monclient(hunting): authenticate will time out at 793.556129s 2024-03-29T10:46:22.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93affff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f93b014a700 0x7f93b00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:56172/0 (socket says 172.21.15.64:56172) 2024-03-29T10:46:22.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93affff640 1 -- 172.21.15.64:0/153254495 learned_addr learned my addr 172.21.15.64:0/153254495 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:22.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031412214 0 0) 0x7f93b00af150 con 0x7f93b00b0770 2024-03-29T10:46:22.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:22.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93af7fe640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.257+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a0003520 con 0x7f93b00b0770 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137637978 0 0) 0x7f93b00aba70 con 0x7f93b014a700 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient(hunting): my global_id is 4233 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b00af150 con 0x7f93b014a700 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310640814 0 0) 0x7f93b00af150 con 0x7f93b014a700 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 >> v1:172.21.15.64:6790/0 conn(0x7f93b00b0770 legacy=0x7f93b0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 >> v1:172.21.15.174:6789/0 conn(0x7f93b00ac5a0 legacy=0x7f93b0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:22.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 monclient: found mon.noname-a 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00ac9d0 con 0x7f93b014a700 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f939c0015a0 con 0x7f93b014a700 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: dump: 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.244 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93b7196640 5 monclient: authenticate success, global_id 4233 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93b7196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f939c001700 con 0x7f93b014a700 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 1 -- 172.21.15.64:0/153254495 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f939c001ad0 con 0x7f93b014a700 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:22.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93af7fe640 10 monclient: dump: 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93b7196640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.258+0000 7f93b7196640 4 set_mon_vals no callback set 2024-03-29T10:46:22.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.259+0000 7f93b7196640 10 monclient: shutdown 2024-03-29T10:46:22.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.259+0000 7f93b7196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:22.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.259+0000 7f93b7196640 1 -- 172.21.15.64:0/153254495 >> v1:172.21.15.64:6789/0 conn(0x7f93b014a700 legacy=0x7f93b00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.259+0000 7f93b7196640 1 -- 172.21.15.64:0/153254495 shutdown_connections 2024-03-29T10:46:22.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.259+0000 7f93b7196640 1 -- 172.21.15.64:0/153254495 >> 172.21.15.64:0/153254495 conn(0x7f93b006b660 msgr2=0x7f93b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:22.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.260+0000 7f93b7196640 1 -- 172.21.15.64:0/153254495 shutdown_connections 2024-03-29T10:46:22.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.260+0000 7f93b7196640 1 -- 172.21.15.64:0/153254495 wait complete. 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.261+0000 7f93b7196640 10 monclient: build_initial_monmap 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.261+0000 7f93b7196640 10 monclient: monmap: 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.254 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.261+0000 7f93b7196640 1 librados: starting msgr at 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.261+0000 7f93b7196640 1 librados: starting objecter 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.261+0000 7f93b7196640 1 Processor -- start 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.262+0000 7f93b7196640 1 -- start start 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.262+0000 7f93b7196640 1 librados: setting wanted keys 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.262+0000 7f93b7196640 1 librados: calling monclient init 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.262+0000 7f93b7196640 10 monclient: init 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:22.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient(hunting): picked mon.noname-c con 0x7f93b00ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient(hunting): picked mon.noname-a con 0x7f93b00b0770 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient(hunting): picked mon.noname-b con 0x7f93b014a700 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b0128640 con 0x7f93b00b0770 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b0128870 con 0x7f93b014a700 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b01266c0 con 0x7f93b00ac5a0 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93b7196640 10 monclient(hunting): authenticate will time out at 793.562689s 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93affff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f93b00b0770 0x7f93b0126be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:56666/0 (socket says 172.21.15.64:56666) 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.263+0000 7f93affff640 1 -- 172.21.15.64:0/2139265273 learned_addr learned my addr 172.21.15.64:0/2139265273 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 client.?.objecter ms_handle_connect 0x7f93b00b0770 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 client.?.objecter ms_handle_connect 0x7f93b00ac5a0 2024-03-29T10:46:22.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217964828 0 0) 0x7f93b0128640 con 0x7f93b00b0770 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): my global_id is 4236 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9394004eb0 con 0x7f93b00b0770 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001108576 0 0) 0x7f93b01266c0 con 0x7f93b00ac5a0 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b0128640 con 0x7f93b00ac5a0 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 client.?.objecter ms_handle_connect 0x7f93b014a700 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.264+0000 7f93ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:22.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1820062803 0 0) 0x7f9394004eb0 con 0x7f93b00b0770 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93b01266c0 con 0x7f93b00b0770 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f939c001460 con 0x7f93b00b0770 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:22.258 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125728822 0 0) 0x7f93b0128870 con 0x7f93b014a700 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9394004eb0 con 0x7f93b014a700 2024-03-29T10:46:22.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989276962 0 0) 0x7f93b0128640 con 0x7f93b00ac5a0 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93b0128870 con 0x7f93b00ac5a0 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f93a4001950 con 0x7f93b00ac5a0 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:22.260 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1244272031 0 0) 0x7f93b01266c0 con 0x7f93b00b0770 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 >> v1:172.21.15.64:6790/0 conn(0x7f93b00ac5a0 legacy=0x7f93b0145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 >> v1:172.21.15.174:6789/0 conn(0x7f93b014a700 legacy=0x7f93b01272f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 monclient: found mon.a 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b01261a0 con 0x7f93b00b0770 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient: _finish_auth 0 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:46:22.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 5 monclient: authenticate success, global_id 4236 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 10 monclient: _renew_subs 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b0126470 con 0x7f93b00b0770 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 10 monclient: _renew_subs 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b01dfc80 con 0x7f93b00b0770 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f939c004a90 con 0x7f93b00b0770 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93b7196640 1 librados: init done 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.265+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f939c004da0 con 0x7f93b00b0770 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.266+0000 7f93ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:22.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.266+0000 7f93ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.266+0000 7f93ad7fa640 10 monclient: dump: 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.266+0000 7f93aeffd640 4 set_mon_vals no callback set 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.268+0000 7f93b7196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.268+0000 7f93b7196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:22.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.268+0000 7f93b7196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.268+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9390005080 con 0x7f93b00b0770 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.269+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f939c01ad70 con 0x7f93b00b0770 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f939c084b90 con 0x7f93b00b0770 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93ad7fa640 10 client.4236.objecter ms_dispatch 0x7f93b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93ad7fa640 3 client.4236.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93ad7fa640 3 client.4236.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93ad7fa640 20 client.4236.objecter dump_active .. 0 homeless 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93b4f32640 10 monclient: get_auth_request con 0x7f939406dfb0 auth_method 0 2024-03-29T10:46:22.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.270+0000 7f93ad7fa640 10 client.4236.objecter ms_handle_connect 0x7f939406dfb0 2024-03-29T10:46:22.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.278+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f939c050470 con 0x7f93b00b0770 2024-03-29T10:46:22.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.278+0000 7f93ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:22.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.278+0000 7f93ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.681+0000 7f93b7196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.681+0000 7f93b7196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.681+0000 7f93b7196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:22.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.681+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9390005330 con 0x7f93b00b0770 2024-03-29T10:46:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.682+0000 7f93ad7fa640 1 -- 172.21.15.64:0/2139265273 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f939c0523a0 con 0x7f93b00b0770 2024-03-29T10:46:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.682+0000 7f93ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:22.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.682+0000 7f93ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:22.665 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:22.665 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 10 librados: watch_flush enter 2024-03-29T10:46:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 10 librados: watch_flush exit 2024-03-29T10:46:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 20 client.4236.objecter shutdown clearing up homeless session... 2024-03-29T10:46:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 10 client.4236.objecter successfully canceled tick 2024-03-29T10:46:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 >> v1:172.21.15.174:6816/2473791532 conn(0x7f939406dfb0 legacy=0x7f9394070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 10 monclient: shutdown 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 >> v1:172.21.15.64:6789/0 conn(0x7f93b00b0770 legacy=0x7f93b0126be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 shutdown_connections 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 >> 172.21.15.64:0/2139265273 conn(0x7f93b006b660 msgr2=0x7f93b00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.686+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 shutdown_connections 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.687+0000 7f93b7196640 1 -- 172.21.15.64:0/2139265273 wait complete. 2024-03-29T10:46:22.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:22.687+0000 7f93b7196640 1 librados: shutdown 2024-03-29T10:46:22.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:29.492 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:29.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.652+0000 7fc222586640 10 monclient: get_monmap_and_config 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.652+0000 7fc222586640 10 monclient: build_initial_monmap 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.652+0000 7fc222586640 10 monclient: monmap: 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:29.653241+0000 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:29.653241+0000 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:29.635 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:29.636 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.653+0000 7fc222586640 1 Processor -- start 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 1 -- start start 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 10 monclient: init 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 10 monclient(hunting): picked mon.noname-b con 0x7fc214006a70 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 10 monclient(hunting): picked mon.noname-a con 0x7fc21402b100 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:29.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.654+0000 7fc222586640 10 monclient(hunting): picked mon.noname-c con 0x7fc214029c30 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc222586640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21402a270 con 0x7fc21402b100 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc222586640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21402c440 con 0x7fc214029c30 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc222586640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140c9440 con 0x7fc214006a70 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc222586640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc222586640 10 monclient(hunting): authenticate will time out at 800.953885s 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc221d85640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 0x7fc2140d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:44382/0 (socket says 172.21.15.64:44382) 2024-03-29T10:46:29.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc221d85640 1 -- 172.21.15.64:0/3142123154 learned_addr learned my addr 172.21.15.64:0/3142123154 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265576813 0 0) 0x7fc21402c440 con 0x7fc214029c30 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc208003520 con 0x7fc214029c30 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc221d85640 1 -- 172.21.15.64:0/3142123154 >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 legacy=0x7fc2140d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc221d85640 1 -- 172.21.15.64:0/3142123154 >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 legacy=0x7fc2140d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc221d85640 1 --1- 172.21.15.64:0/3142123154 >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 0x7fc2140d74a0 :-1 s=OPENED pgs=69115 cs=1 l=1).handle_message read tag failed 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc221d85640 1 --1- 172.21.15.64:0/3142123154 >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 0x7fc2140d74a0 :-1 s=OPENED pgs=69115 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:46:29.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.655+0000 7fc213fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.174:6789/0 2024-03-29T10:46:29.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043308134 0 0) 0x7fc2140c9440 con 0x7fc214006a70 2024-03-29T10:46:29.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:46:29.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21402c440 con 0x7fc214006a70 2024-03-29T10:46:29.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.658+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679224392 0 0) 0x7fc21402c440 con 0x7fc214006a70 2024-03-29T10:46:29.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 legacy=0x7fc2140d74a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:46:29.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 >> v1:172.21.15.64:6789/0 conn(0x7fc21402b100 legacy=0x7fc2140cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:29.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 1 monclient: found mon.noname-b 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2140c95d0 con 0x7fc214006a70 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc222586640 5 monclient: authenticate success, global_id 4235 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc222586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc20c001690 con 0x7fc214006a70 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: dump: 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc222586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc20c001860 con 0x7fc214006a70 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc213fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:29.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc222586640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.659+0000 7fc222586640 4 set_mon_vals no callback set 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc213fff640 1 -- 172.21.15.64:0/3142123154 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc20c001cb0 con 0x7fc214006a70 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc213fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc213fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc213fff640 10 monclient: dump: 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:29.648 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 10 monclient: shutdown 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 1 -- 172.21.15.64:0/3142123154 >> v1:172.21.15.64:6790/0 conn(0x7fc214006a70 legacy=0x7fc2140c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 1 -- 172.21.15.64:0/3142123154 shutdown_connections 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 1 -- 172.21.15.64:0/3142123154 >> 172.21.15.64:0/3142123154 conn(0x7fc2140058f0 msgr2=0x7fc214005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 1 -- 172.21.15.64:0/3142123154 shutdown_connections 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.660+0000 7fc222586640 1 -- 172.21.15.64:0/3142123154 wait complete. 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.661+0000 7fc222586640 10 monclient: build_initial_monmap 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.661+0000 7fc222586640 10 monclient: monmap: 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:29.649 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 librados: starting msgr at 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 librados: starting objecter 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 Processor -- start 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 -- start start 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 librados: setting wanted keys 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 librados: calling monclient init 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 10 monclient: init 2024-03-29T10:46:29.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 10 monclient(hunting): picked mon.noname-a con 0x7fc214006a70 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 10 monclient(hunting): picked mon.noname-b con 0x7fc214029c30 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 10 monclient(hunting): picked mon.noname-c con 0x7fc214032550 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.662+0000 7fc222586640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21402f540 con 0x7fc214006a70 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.663+0000 7fc222586640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21402d500 con 0x7fc214029c30 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.663+0000 7fc222586640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140caa00 con 0x7fc214032550 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.663+0000 7fc222586640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.663+0000 7fc222586640 10 monclient(hunting): authenticate will time out at 800.961878s 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.663+0000 7fc220d83640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 0x7fc21402da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:44464/0 (socket says 172.21.15.64:44464) 2024-03-29T10:46:29.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.663+0000 7fc220d83640 1 -- 172.21.15.64:0/2431429779 learned_addr learned my addr 172.21.15.64:0/2431429779 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.665+0000 7fc211ffb640 10 client.?.objecter ms_handle_connect 0x7fc214029c30 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.665+0000 7fc211ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.665+0000 7fc211ffb640 10 client.?.objecter ms_handle_connect 0x7fc214006a70 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.665+0000 7fc211ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569816692 0 0) 0x7fc21402d500 con 0x7fc214029c30 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc200004eb0 con 0x7fc214029c30 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 client.?.objecter ms_handle_connect 0x7fc214032550 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792132819 0 0) 0x7fc21402f540 con 0x7fc214006a70 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.666+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21402d500 con 0x7fc214006a70 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2488264458 0 0) 0x7fc200004eb0 con 0x7fc214029c30 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc21402f540 con 0x7fc214029c30 2024-03-29T10:46:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc218003fb0 con 0x7fc214029c30 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 345392153 0 0) 0x7fc21402d500 con 0x7fc214006a70 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc200004eb0 con 0x7fc214006a70 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc20c001b70 con 0x7fc214006a70 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:29.660 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788223651 0 0) 0x7fc2140caa00 con 0x7fc214032550 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.667+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21402d500 con 0x7fc214032550 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311302689 0 0) 0x7fc21402d500 con 0x7fc214032550 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2140caa00 con 0x7fc214032550 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1770487445 0 0) 0x7fc21402f540 con 0x7fc214029c30 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 >> v1:172.21.15.64:6790/0 conn(0x7fc214032550 legacy=0x7fc21416c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 >> v1:172.21.15.64:6789/0 conn(0x7fc214006a70 legacy=0x7fc214031e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 monclient: found mon.b 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214164e00 con 0x7fc214029c30 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2180044c0 con 0x7fc214029c30 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc218001070 con 0x7fc214029c30 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.668+0000 7fc211ffb640 10 monclient: dump: 2024-03-29T10:46:29.662 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.670+0000 7fc2137fe640 4 set_mon_vals no callback set 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 10 monclient: _renew_subs 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:29.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2141650b0 con 0x7fc214029c30 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 10 monclient: _renew_subs 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2141645c0 con 0x7fc214029c30 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.671+0000 7fc222586640 1 librados: init done 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.672+0000 7fc222586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.672+0000 7fc222586640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.672+0000 7fc222586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.672+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2140cdef0 con 0x7fc214029c30 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.673+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fc2180011d0 con 0x7fc214029c30 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.673+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fc218084900 con 0x7fc214029c30 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.673+0000 7fc211ffb640 10 client.4246.objecter ms_dispatch 0x7fc21c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.673+0000 7fc211ffb640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:29.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.673+0000 7fc211ffb640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:29.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.673+0000 7fc211ffb640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T10:46:29.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.674+0000 7fc221584640 10 monclient: get_auth_request con 0x7fc20006df20 auth_method 0 2024-03-29T10:46:29.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.674+0000 7fc211ffb640 10 client.4246.objecter ms_handle_connect 0x7fc20006df20 2024-03-29T10:46:29.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.682+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2180501e0 con 0x7fc214029c30 2024-03-29T10:46:29.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.682+0000 7fc211ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:29.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:29.682+0000 7fc211ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:29.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.016+0000 7fc222586640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.016+0000 7fc222586640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.016+0000 7fc222586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:29.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.016+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc214004cd0 con 0x7fc214029c30 2024-03-29T10:46:30.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.017+0000 7fc211ffb640 1 -- 172.21.15.64:0/2431429779 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fc218052110 con 0x7fc214029c30 2024-03-29T10:46:30.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.017+0000 7fc211ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:30.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.017+0000 7fc211ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:30.000 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:30.000 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:30.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 10 librados: watch_flush enter 2024-03-29T10:46:30.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 10 librados: watch_flush exit 2024-03-29T10:46:30.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T10:46:30.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 10 client.4246.objecter successfully canceled tick 2024-03-29T10:46:30.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 >> v1:172.21.15.174:6816/2473791532 conn(0x7fc20006df20 legacy=0x7fc2000703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:30.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 10 monclient: shutdown 2024-03-29T10:46:30.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:30.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.022+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 >> v1:172.21.15.174:6789/0 conn(0x7fc214029c30 legacy=0x7fc21402da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:30.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.024+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 shutdown_connections 2024-03-29T10:46:30.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.024+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 >> 172.21.15.64:0/2431429779 conn(0x7fc2140058f0 msgr2=0x7fc2140c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:30.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.024+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 shutdown_connections 2024-03-29T10:46:30.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.024+0000 7fc222586640 1 -- 172.21.15.64:0/2431429779 wait complete. 2024-03-29T10:46:30.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:30.024+0000 7fc222586640 1 librados: shutdown 2024-03-29T10:46:30.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:36.832 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:36.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.995+0000 7f32ce715640 10 monclient: get_monmap_and_config 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.995+0000 7f32ce715640 10 monclient: build_initial_monmap 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.995+0000 7f32ce715640 10 monclient: monmap: 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:36.996148+0000 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:36.996148+0000 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:36.978 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:36.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.996+0000 7f32ce715640 1 Processor -- start 2024-03-29T10:46:36.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.996+0000 7f32ce715640 1 -- start start 2024-03-29T10:46:36.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.996+0000 7f32ce715640 10 monclient: init 2024-03-29T10:46:36.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient(hunting): picked mon.noname-a con 0x7f32c806d9d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient(hunting): picked mon.noname-b con 0x7f32c813e820 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient(hunting): picked mon.noname-c con 0x7f32c80af1d0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80aed10 con 0x7f32c806d9d0 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80afd50 con 0x7f32c80af1d0 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80ab490 con 0x7f32c813e820 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.997+0000 7f32ce715640 10 monclient(hunting): authenticate will time out at 808.296232s 2024-03-29T10:46:36.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.998+0000 7f32cccb2640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f32c80af1d0 0x7f32c80af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:41392/0 (socket says 172.21.15.64:41392) 2024-03-29T10:46:36.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:36.998+0000 7f32cccb2640 1 -- 172.21.15.64:0/464033490 learned_addr learned my addr 172.21.15.64:0/464033490 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:36.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033060336 0 0) 0x7f32c80ab490 con 0x7f32c813e820 2024-03-29T10:46:36.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:46:36.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b0003520 con 0x7f32c813e820 2024-03-29T10:46:36.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062300688 0 0) 0x7f32c80afd50 con 0x7f32c80af1d0 2024-03-29T10:46:36.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:46:36.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b00048a0 con 0x7f32c80af1d0 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233500360 0 0) 0x7f32b0003520 con 0x7f32c813e820 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 >> v1:172.21.15.174:6789/0 conn(0x7f32c80af1d0 legacy=0x7f32c80af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 >> v1:172.21.15.64:6789/0 conn(0x7f32c806d9d0 legacy=0x7f32c80b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c80ab770 con 0x7f32c813e820 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.000+0000 7f32c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32ce715640 5 monclient: authenticate success, global_id 4241 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32ce715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:36.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32b8001670 con 0x7f32c813e820 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32c6ffd640 10 monclient: dump: 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:36.986 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32ce715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32c6ffd640 1 -- 172.21.15.64:0/464033490 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32b8001850 con 0x7f32c813e820 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32ce715640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.001+0000 7f32ce715640 4 set_mon_vals no callback set 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.002+0000 7f32ce715640 10 monclient: shutdown 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.002+0000 7f32ce715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.002+0000 7f32ce715640 1 -- 172.21.15.64:0/464033490 >> v1:172.21.15.64:6790/0 conn(0x7f32c813e820 legacy=0x7f32c813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.005+0000 7f32ce715640 1 -- 172.21.15.64:0/464033490 shutdown_connections 2024-03-29T10:46:36.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.005+0000 7f32ce715640 1 -- 172.21.15.64:0/464033490 >> 172.21.15.64:0/464033490 conn(0x7f32c806b660 msgr2=0x7f32c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:36.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.007+0000 7f32ce715640 1 -- 172.21.15.64:0/464033490 shutdown_connections 2024-03-29T10:46:36.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.008+0000 7f32ce715640 1 -- 172.21.15.64:0/464033490 wait complete. 2024-03-29T10:46:36.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.008+0000 7f32ce715640 10 monclient: build_initial_monmap 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.008+0000 7f32ce715640 10 monclient: monmap: 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.009+0000 7f32ce715640 1 librados: starting msgr at 2024-03-29T10:46:36.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.009+0000 7f32ce715640 1 librados: starting objecter 2024-03-29T10:46:36.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.009+0000 7f32ce715640 1 Processor -- start 2024-03-29T10:46:36.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 1 -- start start 2024-03-29T10:46:36.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 1 librados: setting wanted keys 2024-03-29T10:46:36.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 1 librados: calling monclient init 2024-03-29T10:46:36.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient: init 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient(hunting): picked mon.noname-a con 0x7f32c806d9d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient(hunting): picked mon.noname-c con 0x7f32c80af1d0 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient(hunting): picked mon.noname-b con 0x7f32c813e820 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c8070770 con 0x7f32c806d9d0 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c814c4f0 con 0x7f32c813e820 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c806fc00 con 0x7f32c80af1d0 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:36.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.010+0000 7f32ce715640 10 monclient(hunting): authenticate will time out at 808.309692s 2024-03-29T10:46:36.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.013+0000 7f32cccb2640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f32c813e820 0x7f32c814bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:41784/0 (socket says 172.21.15.64:41784) 2024-03-29T10:46:36.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.014+0000 7f32cccb2640 1 -- 172.21.15.64:0/401802344 learned_addr learned my addr 172.21.15.64:0/401802344 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 client.?.objecter ms_handle_connect 0x7f32c80af1d0 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939339611 0 0) 0x7f32c806fc00 con 0x7f32c80af1d0 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:46:36.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b4004eb0 con 0x7f32c80af1d0 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175131268 0 0) 0x7f32b4004eb0 con 0x7f32c80af1d0 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32c806fc00 con 0x7f32c80af1d0 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32b8001d80 con 0x7f32c80af1d0 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:36.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:36.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.015+0000 7f32c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:37.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 10 client.?.objecter ms_handle_connect 0x7f32c813e820 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2627000685 0 0) 0x7f32c806fc00 con 0x7f32c80af1d0 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 >> v1:172.21.15.174:6789/0 conn(0x7f32c813e820 legacy=0x7f32c814bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 >> v1:172.21.15.64:6789/0 conn(0x7f32c806d9d0 legacy=0x7f32c8070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 1 monclient: found mon.c 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c806fee0 con 0x7f32c80af1d0 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:37.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 5 monclient: authenticate success, global_id 4244 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 10 monclient: _renew_subs 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c8070300 con 0x7f32c80af1d0 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 10 monclient: _renew_subs 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c81385d0 con 0x7f32c80af1d0 2024-03-29T10:46:37.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.017+0000 7f32ce715640 1 librados: init done 2024-03-29T10:46:37.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32b80011d0 con 0x7f32c80af1d0 2024-03-29T10:46:37.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:37.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32b8003c00 con 0x7f32c80af1d0 2024-03-29T10:46:37.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:37.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:37.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c4ff9640 10 monclient: dump: 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.019+0000 7f32c67fc640 4 set_mon_vals no callback set 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.020+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f32b8003e20 con 0x7f32c80af1d0 2024-03-29T10:46:37.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.020+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f32b8083b50 con 0x7f32c80af1d0 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.020+0000 7f32c4ff9640 10 client.4244.objecter ms_dispatch 0x7f32c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.020+0000 7f32c4ff9640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.020+0000 7f32c4ff9640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.020+0000 7f32c4ff9640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.021+0000 7f32c7fff640 10 monclient: get_auth_request con 0x7f32b406dcf0 auth_method 0 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.022+0000 7f32c4ff9640 10 client.4244.objecter ms_handle_connect 0x7f32b406dcf0 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.024+0000 7f32ce715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.024+0000 7f32ce715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:37.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.024+0000 7f32ce715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:37.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.024+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a8005080 con 0x7f32c80af1d0 2024-03-29T10:46:37.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.033+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32b804f2c0 con 0x7f32c80af1d0 2024-03-29T10:46:37.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.033+0000 7f32c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:37.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.033+0000 7f32c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.361+0000 7f32ce715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.361+0000 7f32ce715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.361+0000 7f32ce715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:46:37.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.361+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32a8005330 con 0x7f32c80af1d0 2024-03-29T10:46:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.362+0000 7f32c4ff9640 1 -- 172.21.15.64:0/401802344 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f32b8083200 con 0x7f32c80af1d0 2024-03-29T10:46:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.362+0000 7f32c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:37.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.362+0000 7f32c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:37.345 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:37.345 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:37.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.366+0000 7f32ce715640 10 librados: watch_flush enter 2024-03-29T10:46:37.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 10 librados: watch_flush exit 2024-03-29T10:46:37.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T10:46:37.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 10 client.4244.objecter successfully canceled tick 2024-03-29T10:46:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 >> v1:172.21.15.174:6816/2473791532 conn(0x7f32b406dcf0 legacy=0x7f32b40701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:37.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 10 monclient: shutdown 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 >> v1:172.21.15.64:6790/0 conn(0x7f32c80af1d0 legacy=0x7f32c814b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 shutdown_connections 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.367+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 >> 172.21.15.64:0/401802344 conn(0x7f32c806b660 msgr2=0x7f32c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.368+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 shutdown_connections 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.368+0000 7f32ce715640 1 -- 172.21.15.64:0/401802344 wait complete. 2024-03-29T10:46:37.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:37.368+0000 7f32ce715640 1 librados: shutdown 2024-03-29T10:46:37.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:44.177 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:44.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.351+0000 7fd735c7b640 10 monclient: get_monmap_and_config 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.351+0000 7fd735c7b640 10 monclient: build_initial_monmap 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.351+0000 7fd735c7b640 10 monclient: monmap: 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:44.352472+0000 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:44.352472+0000 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:44.334 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:44.335 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.354+0000 7fd735c7b640 1 Processor -- start 2024-03-29T10:46:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.354+0000 7fd735c7b640 1 -- start start 2024-03-29T10:46:44.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.354+0000 7fd735c7b640 10 monclient: init 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7300b4330 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7301470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7301439b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73006f2a0 con 0x7fd7301439b0 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd730146b90 con 0x7fd7301470b0 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7300b2ca0 con 0x7fd7300b4330 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:44.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.355+0000 7fd735c7b640 10 monclient(hunting): authenticate will time out at 815.654198s 2024-03-29T10:46:44.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.358+0000 7fd72effd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fd7301470b0 0x7fd7300b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:58799/0 (socket says 172.21.15.64:58799) 2024-03-29T10:46:44.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.358+0000 7fd72effd640 1 -- 172.21.15.64:0/1259519302 learned_addr learned my addr 172.21.15.64:0/1259519302 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:44.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.358+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632177015 0 0) 0x7fd73006f2a0 con 0x7fd7301439b0 2024-03-29T10:46:44.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:44.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd718003520 con 0x7fd7301439b0 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1967626270 0 0) 0x7fd718003520 con 0x7fd7301439b0 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 >> v1:172.21.15.64:6790/0 conn(0x7fd7300b4330 legacy=0x7fd7300b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 >> v1:172.21.15.174:6789/0 conn(0x7fd7301470b0 legacy=0x7fd7300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 monclient: found mon.noname-a 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73006e0c0 con 0x7fd7301439b0 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:44.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd724001690 con 0x7fd7301439b0 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 10 monclient: dump: 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:44.346 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd735c7b640 5 monclient: authenticate success, global_id 4242 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd735c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.359+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd724001860 con 0x7fd7301439b0 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.360+0000 7fd72e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.360+0000 7fd72e7fc640 1 -- 172.21.15.64:0/1259519302 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd724001cb0 con 0x7fd7301439b0 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.360+0000 7fd72e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.360+0000 7fd72e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.360+0000 7fd72e7fc640 10 monclient: dump: 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.347 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.362+0000 7fd735c7b640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:44.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.362+0000 7fd735c7b640 4 set_mon_vals no callback set 2024-03-29T10:46:44.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 10 monclient: shutdown 2024-03-29T10:46:44.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 1 -- 172.21.15.64:0/1259519302 >> v1:172.21.15.64:6789/0 conn(0x7fd7301439b0 legacy=0x7fd730151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 1 -- 172.21.15.64:0/1259519302 shutdown_connections 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 1 -- 172.21.15.64:0/1259519302 >> 172.21.15.64:0/1259519302 conn(0x7fd730066a00 msgr2=0x7fd730066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 1 -- 172.21.15.64:0/1259519302 shutdown_connections 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.363+0000 7fd735c7b640 1 -- 172.21.15.64:0/1259519302 wait complete. 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.364+0000 7fd735c7b640 10 monclient: build_initial_monmap 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.364+0000 7fd735c7b640 10 monclient: monmap: 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:44.350 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.364+0000 7fd735c7b640 1 librados: starting msgr at 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.364+0000 7fd735c7b640 1 librados: starting objecter 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.364+0000 7fd735c7b640 1 Processor -- start 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.365+0000 7fd735c7b640 1 -- start start 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.365+0000 7fd735c7b640 1 librados: setting wanted keys 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.365+0000 7fd735c7b640 1 librados: calling monclient init 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.365+0000 7fd735c7b640 10 monclient: init 2024-03-29T10:46:44.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.365+0000 7fd735c7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:44.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.365+0000 7fd735c7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:44.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.369+0000 7fd735c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7300b4330 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:44.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.369+0000 7fd72f7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd7300b4330 0x7fd7301db3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33185/0 (socket says 172.21.15.64:33185) 2024-03-29T10:46:44.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.369+0000 7fd72f7fe640 1 -- 172.21.15.64:0/3049639055 learned_addr learned my addr 172.21.15.64:0/3049639055 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:44.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.369+0000 7fd717fff640 10 client.?.objecter ms_handle_connect 0x7fd7300b4330 2024-03-29T10:46:44.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.369+0000 7fd717fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:44.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.370+0000 7fd735c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7301439b0 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:44.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.372+0000 7fd735c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7301470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd730121260 con 0x7fd7300b4330 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 client.?.objecter ms_handle_connect 0x7fd7301470b0 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd730141e80 con 0x7fd7301470b0 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7301216f0 con 0x7fd7301439b0 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461989491 0 0) 0x7fd730121260 con 0x7fd7300b4330 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd735c7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd735c7b640 10 monclient(hunting): authenticate will time out at 815.672562s 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:44.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd71c004eb0 con 0x7fd7300b4330 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265637325 0 0) 0x7fd730141e80 con 0x7fd7301470b0 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.373+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd730121260 con 0x7fd7301470b0 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.374+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 689624287 0 0) 0x7fd730121260 con 0x7fd7301470b0 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.374+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd730141e80 con 0x7fd7301470b0 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.374+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd724001b70 con 0x7fd7301470b0 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.374+0000 7fd717fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.374+0000 7fd717fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:44.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.374+0000 7fd717fff640 10 monclient(hunting): dump: 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:44.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.375+0000 7fd717fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:44.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838215529 0 0) 0x7fd730141e80 con 0x7fd7301470b0 2024-03-29T10:46:44.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 >> v1:172.21.15.64:6790/0 conn(0x7fd7301439b0 legacy=0x7fd730141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 >> v1:172.21.15.64:6789/0 conn(0x7fd7300b4330 legacy=0x7fd7301db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:44.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 1 monclient: found mon.b 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd730121880 con 0x7fd7301470b0 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.376+0000 7fd717fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd724001d50 con 0x7fd7301470b0 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd717fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd724003ae0 con 0x7fd7301470b0 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd717fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd717fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd717fff640 10 monclient: dump: 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:44.363 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd735c7b640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd735c7b640 10 monclient: _renew_subs 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd735c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.377+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd730142680 con 0x7fd7301470b0 2024-03-29T10:46:44.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.378+0000 7fd735c7b640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:44.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.378+0000 7fd735c7b640 10 monclient: _renew_subs 2024-03-29T10:46:44.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.378+0000 7fd735c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:44.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.378+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7301dbcc0 con 0x7fd7301470b0 2024-03-29T10:46:44.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.378+0000 7fd735c7b640 1 librados: init done 2024-03-29T10:46:44.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.378+0000 7fd72dffb640 4 set_mon_vals no callback set 2024-03-29T10:46:44.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.379+0000 7fd735c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:44.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.380+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fd724019ab0 con 0x7fd7301470b0 2024-03-29T10:46:44.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fd7240838a0 con 0x7fd7301470b0 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd717fff640 10 client.4255.objecter ms_dispatch 0x7fd730000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd717fff640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd717fff640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd717fff640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd735c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd735c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.381+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7301227d0 con 0x7fd7301470b0 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.383+0000 7fd72f7fe640 10 monclient: get_auth_request con 0x7fd71c06dd10 auth_method 0 2024-03-29T10:46:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.384+0000 7fd717fff640 10 client.4255.objecter ms_handle_connect 0x7fd71c06dd10 2024-03-29T10:46:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.390+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd72404f180 con 0x7fd7301470b0 2024-03-29T10:46:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.390+0000 7fd717fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.390+0000 7fd717fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:44.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.721+0000 7fd735c7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.721+0000 7fd735c7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.721+0000 7fd735c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:44.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.721+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd730075f10 con 0x7fd7301470b0 2024-03-29T10:46:44.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.722+0000 7fd717fff640 1 -- 172.21.15.64:0/3049639055 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fd7240510b0 con 0x7fd7301470b0 2024-03-29T10:46:44.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.722+0000 7fd717fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.722+0000 7fd717fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:44.714 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:44.714 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:44.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 10 librados: watch_flush enter 2024-03-29T10:46:44.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 10 librados: watch_flush exit 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 10 client.4255.objecter successfully canceled tick 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 >> v1:172.21.15.174:6816/2473791532 conn(0x7fd71c06dd10 legacy=0x7fd71c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 10 monclient: shutdown 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.729+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 >> v1:172.21.15.174:6789/0 conn(0x7fd7301470b0 legacy=0x7fd730141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.731+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 shutdown_connections 2024-03-29T10:46:44.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.731+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 >> 172.21.15.64:0/3049639055 conn(0x7fd730066a00 msgr2=0x7fd7301d1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:44.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.733+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 shutdown_connections 2024-03-29T10:46:44.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.733+0000 7fd735c7b640 1 -- 172.21.15.64:0/3049639055 wait complete. 2024-03-29T10:46:44.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:44.733+0000 7fd735c7b640 1 librados: shutdown 2024-03-29T10:46:44.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:51.544 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:51.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.710+0000 7fdd639f6640 10 monclient: get_monmap_and_config 2024-03-29T10:46:51.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.710+0000 7fdd639f6640 10 monclient: build_initial_monmap 2024-03-29T10:46:51.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.710+0000 7fdd639f6640 10 monclient: monmap: 2024-03-29T10:46:51.697 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:51.697 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:51.712630+0000 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:51.712630+0000 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.713+0000 7fdd639f6640 1 Processor -- start 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.713+0000 7fdd639f6640 1 -- start start 2024-03-29T10:46:51.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.713+0000 7fdd639f6640 10 monclient: init 2024-03-29T10:46:51.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:51.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:51.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd5c0afb20 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:51.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd5c0ad7c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:51.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd5c13d720 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:51.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0aeb10 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0aedf0 con 0x7fdd5c13d720 2024-03-29T10:46:51.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0af0d0 con 0x7fdd5c0afb20 2024-03-29T10:46:51.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:51.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.714+0000 7fdd639f6640 10 monclient(hunting): authenticate will time out at 823.014728s 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.718+0000 7fdd61792640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fdd5c0afb20 0x7fdd5c13ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50621/0 (socket says 172.21.15.64:50621) 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.718+0000 7fdd61792640 1 -- 172.21.15.64:0/3262990600 learned_addr learned my addr 172.21.15.64:0/3262990600 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.719+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95683426 0 0) 0x7fdd5c0af0d0 con 0x7fdd5c0afb20 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.719+0000 7fdd53fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.719+0000 7fdd53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.719+0000 7fdd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.719+0000 7fdd53fff640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:46:51.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.719+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd48003520 con 0x7fdd5c0afb20 2024-03-29T10:46:51.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198731858 0 0) 0x7fdd48003520 con 0x7fdd5c0afb20 2024-03-29T10:46:51.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 >> v1:172.21.15.174:6789/0 conn(0x7fdd5c13d720 legacy=0x7fdd5c14b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:46:51.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 >> v1:172.21.15.64:6789/0 conn(0x7fdd5c0ad7c0 legacy=0x7fdd5c13d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:51.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 monclient: found mon.noname-b 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0af3b0 con 0x7fdd5c0afb20 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:51.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd639f6640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd639f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdd4c001690 con 0x7fdd5c0afb20 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: dump: 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.709 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd639f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 1 -- 172.21.15.64:0/3262990600 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd4c001920 con 0x7fdd5c0afb20 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd639f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.722+0000 7fdd639f6640 4 set_mon_vals no callback set 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.723+0000 7fdd639f6640 10 monclient: shutdown 2024-03-29T10:46:51.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.723+0000 7fdd639f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:51.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.723+0000 7fdd639f6640 1 -- 172.21.15.64:0/3262990600 >> v1:172.21.15.64:6790/0 conn(0x7fdd5c0afb20 legacy=0x7fdd5c13ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.723+0000 7fdd639f6640 1 -- 172.21.15.64:0/3262990600 shutdown_connections 2024-03-29T10:46:51.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.723+0000 7fdd639f6640 1 -- 172.21.15.64:0/3262990600 >> 172.21.15.64:0/3262990600 conn(0x7fdd5c0b35e0 msgr2=0x7fdd5c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.725+0000 7fdd639f6640 1 -- 172.21.15.64:0/3262990600 shutdown_connections 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.726+0000 7fdd639f6640 1 -- 172.21.15.64:0/3262990600 wait complete. 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 10 monclient: build_initial_monmap 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 10 monclient: monmap: 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.714 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 1 librados: starting msgr at 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 1 librados: starting objecter 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 1 Processor -- start 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 1 -- start start 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 1 librados: setting wanted keys 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 1 librados: calling monclient init 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.727+0000 7fdd639f6640 10 monclient: init 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.728+0000 7fdd639f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:51.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.728+0000 7fdd639f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd5c0ad7c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd5c0afb20 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd5c13d720 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0b19b0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c1d6ee0 con 0x7fdd5c0afb20 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c1d7070 con 0x7fdd5c13d720 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.729+0000 7fdd639f6640 10 monclient(hunting): authenticate will time out at 823.029508s 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.733+0000 7fdd61792640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fdd5c0ad7c0 0x7fdd5c1d67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40813/0 (socket says 172.21.15.64:40813) 2024-03-29T10:46:51.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.733+0000 7fdd61792640 1 -- 172.21.15.64:0/140820781 learned_addr learned my addr 172.21.15.64:0/140820781 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:51.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.733+0000 7fdd51ffb640 10 client.?.objecter ms_handle_connect 0x7fdd5c0ad7c0 2024-03-29T10:46:51.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.733+0000 7fdd51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 client.?.objecter ms_handle_connect 0x7fdd5c13d720 2024-03-29T10:46:51.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557000593 0 0) 0x7fdd5c0b19b0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:46:51.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd40004eb0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 client.?.objecter ms_handle_connect 0x7fdd5c0afb20 2024-03-29T10:46:51.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.735+0000 7fdd51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255743788 0 0) 0x7fdd5c1d7070 con 0x7fdd5c13d720 2024-03-29T10:46:51.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5c0b19b0 con 0x7fdd5c13d720 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602351025 0 0) 0x7fdd5c1d6ee0 con 0x7fdd5c0afb20 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:46:51.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.738+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd40005360 con 0x7fdd5c0afb20 2024-03-29T10:46:51.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 412260007 0 0) 0x7fdd40004eb0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd400057b0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdd4c004ae0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:51.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 21382307 0 0) 0x7fdd5c0b19b0 con 0x7fdd5c13d720 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd40004eb0 con 0x7fdd5c13d720 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068517395 0 0) 0x7fdd40005360 con 0x7fdd5c0afb20 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd40008500 con 0x7fdd5c0afb20 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdd580016f0 con 0x7fdd5c13d720 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.725 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdd54001890 con 0x7fdd5c0afb20 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.726 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2685232382 0 0) 0x7fdd400057b0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 >> v1:172.21.15.64:6790/0 conn(0x7fdd5c13d720 legacy=0x7fdd5c1dd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 >> v1:172.21.15.174:6789/0 conn(0x7fdd5c0afb20 legacy=0x7fdd5c0b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 monclient: found mon.a 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:51.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c13ddb0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd639f6640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd639f6640 10 monclient: _renew_subs 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd639f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.739+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0b2b10 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.740+0000 7fdd639f6640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.740+0000 7fdd639f6640 10 monclient: _renew_subs 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.740+0000 7fdd639f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:51.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.740+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c1dee10 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.740+0000 7fdd639f6640 1 librados: init done 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.743+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd4c001b10 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.743+0000 7fdd51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.743+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdd4c005090 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.743+0000 7fdd51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.743+0000 7fdd51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.743+0000 7fdd51ffb640 10 monclient: dump: 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.731 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.744+0000 7fdd537fe640 4 set_mon_vals no callback set 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.744+0000 7fdd639f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.745+0000 7fdd639f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.745+0000 7fdd639f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.745+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c005080 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.746+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fdd4c005270 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.747+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fdd4c084ea0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.747+0000 7fdd51ffb640 10 client.4248.objecter ms_dispatch 0x7fdd5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:51.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.747+0000 7fdd51ffb640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:51.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.747+0000 7fdd51ffb640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:51.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.747+0000 7fdd51ffb640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T10:46:51.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.748+0000 7fdd60f91640 10 monclient: get_auth_request con 0x7fdd4006e800 auth_method 0 2024-03-29T10:46:51.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.750+0000 7fdd51ffb640 10 client.4248.objecter ms_handle_connect 0x7fdd4006e800 2024-03-29T10:46:51.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.758+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd4c050780 con 0x7fdd5c0ad7c0 2024-03-29T10:46:51.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.758+0000 7fdd51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:51.758+0000 7fdd51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.082+0000 7fdd639f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.082+0000 7fdd639f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.082+0000 7fdd639f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:46:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.082+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd3c005330 con 0x7fdd5c0ad7c0 2024-03-29T10:46:52.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.086+0000 7fdd51ffb640 1 -- 172.21.15.64:0/140820781 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fdd4c0526b0 con 0x7fdd5c0ad7c0 2024-03-29T10:46:52.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.086+0000 7fdd51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:52.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.086+0000 7fdd51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:52.070 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:52.070 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:52.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.091+0000 7fdd639f6640 10 librados: watch_flush enter 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 10 librados: watch_flush exit 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 10 client.4248.objecter successfully canceled tick 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 >> v1:172.21.15.174:6816/2473791532 conn(0x7fdd4006e800 legacy=0x7fdd40070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 10 monclient: shutdown 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 >> v1:172.21.15.64:6789/0 conn(0x7fdd5c0ad7c0 legacy=0x7fdd5c1d67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 shutdown_connections 2024-03-29T10:46:52.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.092+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 >> 172.21.15.64:0/140820781 conn(0x7fdd5c0b35e0 msgr2=0x7fdd5c1369d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:52.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.093+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 shutdown_connections 2024-03-29T10:46:52.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.093+0000 7fdd639f6640 1 -- 172.21.15.64:0/140820781 wait complete. 2024-03-29T10:46:52.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:52.093+0000 7fdd639f6640 1 librados: shutdown 2024-03-29T10:46:52.093 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:58.900 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.071+0000 7f14442c9640 10 monclient: get_monmap_and_config 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.071+0000 7f14442c9640 10 monclient: build_initial_monmap 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.071+0000 7f14442c9640 10 monclient: monmap: 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:46:59.073673+0000 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:46:59.073673+0000 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:59.055 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:46:59.056 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:46:59.056 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:59.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.073+0000 7f14442c9640 1 Processor -- start 2024-03-29T10:46:59.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.073+0000 7f14442c9640 1 -- start start 2024-03-29T10:46:59.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.073+0000 7f14442c9640 10 monclient: init 2024-03-29T10:46:59.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.074+0000 7f14442c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:59.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.074+0000 7f14442c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:59.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.074+0000 7f14442c9640 10 monclient(hunting): picked mon.noname-b con 0x7f143c0b0500 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:59.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.074+0000 7f14442c9640 10 monclient(hunting): picked mon.noname-c con 0x7f143c0b08e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:59.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441864640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f143c0b08e0 0x7f143c13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:56673/0 (socket says 172.21.15.64:56673) 2024-03-29T10:46:59.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f14442c9640 10 monclient(hunting): picked mon.noname-a con 0x7f143c0aeca0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:59.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f14442c9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c140620 con 0x7f143c0aeca0 2024-03-29T10:46:59.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f14442c9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c13daf0 con 0x7f143c0b08e0 2024-03-29T10:46:59.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f14442c9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c0ae180 con 0x7f143c0b0500 2024-03-29T10:46:59.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f14442c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:59.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f14442c9640 10 monclient(hunting): authenticate will time out at 830.380079s 2024-03-29T10:46:59.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441864640 1 -- 172.21.15.64:0/1336899315 learned_addr learned my addr 172.21.15.64:0/1336899315 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:59.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017244908 0 0) 0x7f143c0ae180 con 0x7f143c0b0500 2024-03-29T10:46:59.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441063640 10 monclient(hunting): _negotiate 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441063640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441063640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441063640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.080+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1428003520 con 0x7f143c0b0500 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580526554 0 0) 0x7f143c140620 con 0x7f143c0aeca0 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient(hunting): _negotiate 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:59.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143c0ae180 con 0x7f143c0aeca0 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246861177 0 0) 0x7f143c0ae180 con 0x7f143c0aeca0 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 >> v1:172.21.15.64:6790/0 conn(0x7f143c0b0500 legacy=0x7f143c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 >> v1:172.21.15.174:6789/0 conn(0x7f143c0b08e0 legacy=0x7f143c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: _finish_hunting 0 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 monclient: found mon.noname-a 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c0accf0 con 0x7f143c0aeca0 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: _finish_auth 0 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: _check_auth_tickets 2024-03-29T10:46:59.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f14340019f0 con 0x7f143c0aeca0 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: dump: 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:59.071 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f14442c9640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f14442c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1434001c10 con 0x7f143c0aeca0 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 1 -- 172.21.15.64:0/1336899315 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1434001fa0 con 0x7f143c0aeca0 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f1441063640 10 monclient: dump: 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.072 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f14442c9640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.081+0000 7f14442c9640 4 set_mon_vals no callback set 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 10 monclient: shutdown 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 1 -- 172.21.15.64:0/1336899315 >> v1:172.21.15.64:6789/0 conn(0x7f143c0aeca0 legacy=0x7f143c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:59.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 1 -- 172.21.15.64:0/1336899315 shutdown_connections 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 1 -- 172.21.15.64:0/1336899315 >> 172.21.15.64:0/1336899315 conn(0x7f143c06b660 msgr2=0x7f143c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 1 -- 172.21.15.64:0/1336899315 shutdown_connections 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.082+0000 7f14442c9640 1 -- 172.21.15.64:0/1336899315 wait complete. 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.083+0000 7f14442c9640 10 monclient: build_initial_monmap 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.083+0000 7f14442c9640 10 monclient: monmap: 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:59.074 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:46:59.075 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:46:59.075 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:46:59.075 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:59.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 1 librados: starting msgr at 2024-03-29T10:46:59.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 1 librados: starting objecter 2024-03-29T10:46:59.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 1 Processor -- start 2024-03-29T10:46:59.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 1 -- start start 2024-03-29T10:46:59.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 1 librados: setting wanted keys 2024-03-29T10:46:59.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 1 librados: calling monclient init 2024-03-29T10:46:59.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.084+0000 7f14442c9640 10 monclient: init 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.085+0000 7f14442c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.085+0000 7f14442c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.085+0000 7f14442c9640 10 monclient(hunting): picked mon.noname-a con 0x7f143c0aeca0 addr v1:172.21.15.64:6789/0 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.085+0000 7f14442c9640 10 monclient(hunting): picked mon.noname-c con 0x7f143c0b0500 addr v1:172.21.15.64:6790/0 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.086+0000 7f14442c9640 10 monclient(hunting): picked mon.noname-b con 0x7f143c136130 addr v1:172.21.15.174:6789/0 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.086+0000 7f14442c9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c137fd0 con 0x7f143c0aeca0 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.086+0000 7f14442c9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c1397c0 con 0x7f143c136130 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.086+0000 7f14442c9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c139aa0 con 0x7f143c0b0500 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.086+0000 7f14442c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.086+0000 7f14442c9640 10 monclient(hunting): authenticate will time out at 830.386286s 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.090+0000 7f1442065640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f143c0aeca0 0x7f143c138460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37903/0 (socket says 172.21.15.64:37903) 2024-03-29T10:46:59.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.090+0000 7f1442065640 1 -- 172.21.15.64:0/1190066688 learned_addr learned my addr 172.21.15.64:0/1190066688 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 client.?.objecter ms_handle_connect 0x7f143c0aeca0 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 client.?.objecter ms_handle_connect 0x7f143c136130 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986473837 0 0) 0x7f143c1397c0 con 0x7f143c136130 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.091+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1420004eb0 con 0x7f143c136130 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899466324 0 0) 0x7f143c137fd0 con 0x7f143c0aeca0 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:59.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143c1397c0 con 0x7f143c0aeca0 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 client.?.objecter ms_handle_connect 0x7f143c0b0500 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106719926 0 0) 0x7f1420004eb0 con 0x7f143c136130 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1420005490 con 0x7f143c136130 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1434001930 con 0x7f143c136130 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:59.082 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510976445 0 0) 0x7f143c139aa0 con 0x7f143c0b0500 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:46:59.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.092+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1420004eb0 con 0x7f143c0b0500 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3990451346 0 0) 0x7f1420005490 con 0x7f143c136130 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 >> v1:172.21.15.64:6790/0 conn(0x7f143c0b0500 legacy=0x7f143c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 >> v1:172.21.15.64:6789/0 conn(0x7f143c0aeca0 legacy=0x7f143c138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 monclient: found mon.b 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c1f2760 con 0x7f143c136130 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:59.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f14442c9640 5 monclient: authenticate success, global_id 4261 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f14442c9640 10 monclient: _renew_subs 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f14442c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c1ef730 con 0x7f143c136130 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1434003c20 con 0x7f143c136130 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1434003f30 con 0x7f143c136130 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1432ffd640 10 monclient: dump: 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:46:59.085 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.094+0000 7f1440862640 4 set_mon_vals no callback set 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.095+0000 7f14442c9640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.095+0000 7f14442c9640 10 monclient: _renew_subs 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.095+0000 7f14442c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.095+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c1efc10 con 0x7f143c136130 2024-03-29T10:46:59.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.095+0000 7f14442c9640 1 librados: init done 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.097+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f1434019f00 con 0x7f143c136130 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.097+0000 7f14442c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.097+0000 7f14442c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.097+0000 7f14442c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.097+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c0ad0c0 con 0x7f143c136130 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.097+0000 7f1442065640 10 monclient: get_auth_request con 0x7f142006e050 auth_method 0 2024-03-29T10:46:59.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.098+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f1434053a20 con 0x7f143c136130 2024-03-29T10:46:59.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.098+0000 7f1432ffd640 10 client.4261.objecter ms_dispatch 0x7f143c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:59.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.098+0000 7f1432ffd640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:59.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.098+0000 7f1432ffd640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:59.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.098+0000 7f1432ffd640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T10:46:59.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.099+0000 7f1432ffd640 10 client.4261.objecter ms_handle_connect 0x7f142006e050 2024-03-29T10:46:59.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.105+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f143404f530 con 0x7f143c136130 2024-03-29T10:46:59.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.105+0000 7f1432ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:59.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.105+0000 7f1432ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:59.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.432+0000 7f14442c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:59.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.432+0000 7f14442c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:59.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.432+0000 7f14442c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:46:59.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.432+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f143c075f10 con 0x7f143c136130 2024-03-29T10:46:59.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.435+0000 7f1432ffd640 1 -- 172.21.15.64:0/1190066688 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f1434051460 con 0x7f143c136130 2024-03-29T10:46:59.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.435+0000 7f1432ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:59.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.435+0000 7f1432ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:59.419 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:46:59.419 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:59.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.438+0000 7f14442c9640 10 librados: watch_flush enter 2024-03-29T10:46:59.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 10 librados: watch_flush exit 2024-03-29T10:46:59.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T10:46:59.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 10 client.4261.objecter successfully canceled tick 2024-03-29T10:46:59.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 >> v1:172.21.15.174:6816/2473791532 conn(0x7f142006e050 legacy=0x7f1420070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:59.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 10 monclient: shutdown 2024-03-29T10:46:59.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:59.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.439+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 >> v1:172.21.15.174:6789/0 conn(0x7f143c136130 legacy=0x7f143c139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:59.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.440+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 shutdown_connections 2024-03-29T10:46:59.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.440+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 >> 172.21.15.64:0/1190066688 conn(0x7f143c06b660 msgr2=0x7f143c1400b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:59.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.441+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 shutdown_connections 2024-03-29T10:46:59.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.442+0000 7f14442c9640 1 -- 172.21.15.64:0/1190066688 wait complete. 2024-03-29T10:46:59.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:46:59.442+0000 7f14442c9640 1 librados: shutdown 2024-03-29T10:46:59.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:06.252 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.422+0000 7fee13513640 10 monclient: get_monmap_and_config 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.422+0000 7fee13513640 10 monclient: build_initial_monmap 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.422+0000 7fee13513640 10 monclient: monmap: 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:06.423631+0000 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:06.423631+0000 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:06.405 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:06.406 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.425+0000 7fee13513640 1 Processor -- start 2024-03-29T10:47:06.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.425+0000 7fee13513640 1 -- start start 2024-03-29T10:47:06.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.425+0000 7fee13513640 10 monclient: init 2024-03-29T10:47:06.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:06.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:06.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:06.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:06.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:06.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0aaef0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c145bd0 con 0x7fee0c1457f0 2024-03-29T10:47:06.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c145eb0 con 0x7fee0c13f650 2024-03-29T10:47:06.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:06.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.426+0000 7fee13513640 10 monclient(hunting): authenticate will time out at 837.725355s 2024-03-29T10:47:06.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.433+0000 7fee112af640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fee0c1457f0 0x7fee0c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:44979/0 (socket says 172.21.15.64:44979) 2024-03-29T10:47:06.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.433+0000 7fee112af640 1 -- 172.21.15.64:0/2923224576 learned_addr learned my addr 172.21.15.64:0/2923224576 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:06.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085787057 0 0) 0x7fee0c145bd0 con 0x7fee0c1457f0 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf8003520 con 0x7fee0c1457f0 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202323271 0 0) 0x7fee0c145eb0 con 0x7fee0c13f650 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:06.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c145bd0 con 0x7fee0c13f650 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292792360 0 0) 0x7fee0c0aaef0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:47:06.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.438+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c145eb0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3579288562 0 0) 0x7fee0c145bd0 con 0x7fee0c13f650 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 >> v1:172.21.15.174:6789/0 conn(0x7fee0c1457f0 legacy=0x7fee0c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 >> v1:172.21.15.64:6789/0 conn(0x7fee0c0a9d70 legacy=0x7fee0c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 monclient: found mon.noname-b 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:06.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c146340 con 0x7fee0c13f650 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fedfc001520 con 0x7fee0c13f650 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: dump: 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:06.425 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee13513640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee13513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedfc001680 con 0x7fee0c13f650 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee13513640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee13513640 4 set_mon_vals no callback set 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 1 -- 172.21.15.64:0/2923224576 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fedfc001ad0 con 0x7fee0c13f650 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:06.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.440+0000 7fee03fff640 10 monclient: dump: 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.441+0000 7fee13513640 10 monclient: shutdown 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.441+0000 7fee13513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:06.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.441+0000 7fee13513640 1 -- 172.21.15.64:0/2923224576 >> v1:172.21.15.64:6790/0 conn(0x7fee0c13f650 legacy=0x7fee0c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.441+0000 7fee13513640 1 -- 172.21.15.64:0/2923224576 shutdown_connections 2024-03-29T10:47:06.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.441+0000 7fee13513640 1 -- 172.21.15.64:0/2923224576 >> 172.21.15.64:0/2923224576 conn(0x7fee0c066a00 msgr2=0x7fee0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:06.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.441+0000 7fee13513640 1 -- 172.21.15.64:0/2923224576 shutdown_connections 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.442+0000 7fee13513640 1 -- 172.21.15.64:0/2923224576 wait complete. 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.442+0000 7fee13513640 10 monclient: build_initial_monmap 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.442+0000 7fee13513640 10 monclient: monmap: 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:06.430 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 1 librados: starting msgr at 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 1 librados: starting objecter 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 1 Processor -- start 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 1 -- start start 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 1 librados: setting wanted keys 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 1 librados: calling monclient init 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.443+0000 7fee13513640 10 monclient: init 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:06.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0ac590 con 0x7fee0c13f650 2024-03-29T10:47:06.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c1e40b0 con 0x7fee0c1457f0 2024-03-29T10:47:06.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c1e4ce0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:06.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.444+0000 7fee13513640 10 monclient(hunting): authenticate will time out at 837.743264s 2024-03-29T10:47:06.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.451+0000 7fee112af640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fee0c0a9d70 0x7fee0c1e45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36121/0 (socket says 172.21.15.64:36121) 2024-03-29T10:47:06.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.451+0000 7fee112af640 1 -- 172.21.15.64:0/2760076670 learned_addr learned my addr 172.21.15.64:0/2760076670 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:06.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 client.?.objecter ms_handle_connect 0x7fee0c1457f0 2024-03-29T10:47:06.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:06.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 client.?.objecter ms_handle_connect 0x7fee0c0a9d70 2024-03-29T10:47:06.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 client.?.objecter ms_handle_connect 0x7fee0c13f650 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367005267 0 0) 0x7fee0c1e40b0 con 0x7fee0c1457f0 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf0004eb0 con 0x7fee0c1457f0 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305923072 0 0) 0x7fee0c1e4ce0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c1e40b0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326501731 0 0) 0x7fee0c1e40b0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c1e4ce0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fee08001970 con 0x7fee0c0a9d70 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:06.442 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.453+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439540737 0 0) 0x7fedf0004eb0 con 0x7fee0c1457f0 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c1e40b0 con 0x7fee0c1457f0 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fee04001bf0 con 0x7fee0c1457f0 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:06.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:06.444 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760513268 0 0) 0x7fee0c0ac590 con 0x7fee0c13f650 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf0004eb0 con 0x7fee0c13f650 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 359397716 0 0) 0x7fee0c1e4ce0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 >> v1:172.21.15.174:6789/0 conn(0x7fee0c1457f0 legacy=0x7fee0c0b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 >> v1:172.21.15.64:6789/0 conn(0x7fee0c13f650 legacy=0x7fee0c071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 monclient: found mon.c 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:06.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c1e4e70 con 0x7fee0c0a9d70 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 10 monclient: _renew_subs 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c071e00 con 0x7fee0c0a9d70 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 10 monclient: _renew_subs 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c06e380 con 0x7fee0c0a9d70 2024-03-29T10:47:06.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.454+0000 7fee13513640 1 librados: init done 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.458+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee08001c70 con 0x7fee0c0a9d70 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.458+0000 7fee01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.458+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fee08004c30 con 0x7fee0c0a9d70 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.458+0000 7fee01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.458+0000 7fee01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.458+0000 7fee01ffb640 10 monclient: dump: 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:06.447 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:06.448 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:06.448 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:06.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.459+0000 7fee13513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:06.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.459+0000 7fee13513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:06.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.459+0000 7fee13513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.459+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c075f10 con 0x7fee0c0a9d70 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.459+0000 7fee02ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.460+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fee0801ac00 con 0x7fee0c0a9d70 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.461+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fee08084a80 con 0x7fee0c0a9d70 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.461+0000 7fee01ffb640 10 client.4262.objecter ms_dispatch 0x7fee0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.461+0000 7fee01ffb640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.461+0000 7fee01ffb640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.461+0000 7fee01ffb640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T10:47:06.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.461+0000 7fee10aae640 10 monclient: get_auth_request con 0x7fedf006df30 auth_method 0 2024-03-29T10:47:06.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.465+0000 7fee01ffb640 10 client.4262.objecter ms_handle_connect 0x7fedf006df30 2024-03-29T10:47:06.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.470+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee080503c0 con 0x7fee0c0a9d70 2024-03-29T10:47:06.452 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.470+0000 7fee01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:06.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.470+0000 7fee01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:06.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.808+0000 7fee13513640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.808+0000 7fee13513640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.808+0000 7fee13513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:06.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.808+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0c1e4240 con 0x7fee0c0a9d70 2024-03-29T10:47:06.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.812+0000 7fee01ffb640 1 -- 172.21.15.64:0/2760076670 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fee08084130 con 0x7fee0c0a9d70 2024-03-29T10:47:06.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.812+0000 7fee01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.812+0000 7fee01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:06.798 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:06.798 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:06.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.819+0000 7fee13513640 10 librados: watch_flush enter 2024-03-29T10:47:06.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.819+0000 7fee13513640 10 librados: watch_flush exit 2024-03-29T10:47:06.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.819+0000 7fee13513640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T10:47:06.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.819+0000 7fee13513640 10 client.4262.objecter successfully canceled tick 2024-03-29T10:47:06.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.819+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 >> v1:172.21.15.174:6816/2473791532 conn(0x7fedf006df30 legacy=0x7fedf00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.819+0000 7fee13513640 10 monclient: shutdown 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 >> v1:172.21.15.64:6790/0 conn(0x7fee0c0a9d70 legacy=0x7fee0c1e45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 shutdown_connections 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 >> 172.21.15.64:0/2760076670 conn(0x7fee0c066a00 msgr2=0x7fee0c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 shutdown_connections 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 1 -- 172.21.15.64:0/2760076670 wait complete. 2024-03-29T10:47:06.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:06.820+0000 7fee13513640 1 librados: shutdown 2024-03-29T10:47:06.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:13.629 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:13.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.805+0000 7f67a3414640 10 monclient: get_monmap_and_config 2024-03-29T10:47:13.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.805+0000 7f67a3414640 10 monclient: build_initial_monmap 2024-03-29T10:47:13.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.805+0000 7f67a3414640 10 monclient: monmap: 2024-03-29T10:47:13.788 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:13.788 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:13.788 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:13.806109+0000 2024-03-29T10:47:13.788 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:13.806109+0000 2024-03-29T10:47:13.788 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:13.788 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:13.789 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:13.789 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:13.789 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:13.789 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:13.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.806+0000 7f67a3414640 1 Processor -- start 2024-03-29T10:47:13.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.806+0000 7f67a3414640 1 -- start start 2024-03-29T10:47:13.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.806+0000 7f67a3414640 10 monclient: init 2024-03-29T10:47:13.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:13.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:13.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient(hunting): picked mon.noname-c con 0x7f679c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:13.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient(hunting): picked mon.noname-a con 0x7f679c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:13.792 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient(hunting): picked mon.noname-b con 0x7f679c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:13.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c0aaef0 con 0x7f679c13f650 2024-03-29T10:47:13.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c145bd0 con 0x7f679c1457f0 2024-03-29T10:47:13.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c145eb0 con 0x7f679c0a9d70 2024-03-29T10:47:13.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:13.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.807+0000 7f67a3414640 10 monclient(hunting): authenticate will time out at 845.106527s 2024-03-29T10:47:13.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.813+0000 7f67a09af640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f679c13f650 0x7f679c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45327/0 (socket says 172.21.15.64:45327) 2024-03-29T10:47:13.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.813+0000 7f67a09af640 1 -- 172.21.15.64:0/2745186260 learned_addr learned my addr 172.21.15.64:0/2745186260 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:13.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.816+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484025848 0 0) 0x7f679c145eb0 con 0x7f679c0a9d70 2024-03-29T10:47:13.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:47:13.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6788003520 con 0x7f679c0a9d70 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647026704 0 0) 0x7f6788003520 con 0x7f679c0a9d70 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 >> v1:172.21.15.174:6789/0 conn(0x7f679c1457f0 legacy=0x7f679c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 >> v1:172.21.15.64:6789/0 conn(0x7f679c13f650 legacy=0x7f679c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 monclient: found mon.noname-b 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c146340 con 0x7f679c0a9d70 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f67a3414640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:47:13.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f67a3414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f678c001690 con 0x7f679c0a9d70 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: dump: 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:13.803 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f67a3414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 1 -- 172.21.15.64:0/2745186260 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f678c001920 con 0x7f679c0a9d70 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f6793fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f67a3414640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.817+0000 7f67a3414640 4 set_mon_vals no callback set 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.818+0000 7f67a3414640 10 monclient: shutdown 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.818+0000 7f67a3414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.818+0000 7f67a3414640 1 -- 172.21.15.64:0/2745186260 >> v1:172.21.15.64:6790/0 conn(0x7f679c0a9d70 legacy=0x7f679c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.818+0000 7f67a3414640 1 -- 172.21.15.64:0/2745186260 shutdown_connections 2024-03-29T10:47:13.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.818+0000 7f67a3414640 1 -- 172.21.15.64:0/2745186260 >> 172.21.15.64:0/2745186260 conn(0x7f679c066a00 msgr2=0x7f679c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:13.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.819+0000 7f67a3414640 1 -- 172.21.15.64:0/2745186260 shutdown_connections 2024-03-29T10:47:13.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.819+0000 7f67a3414640 1 -- 172.21.15.64:0/2745186260 wait complete. 2024-03-29T10:47:13.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.820+0000 7f67a3414640 10 monclient: build_initial_monmap 2024-03-29T10:47:13.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.820+0000 7f67a3414640 10 monclient: monmap: 2024-03-29T10:47:13.807 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 1 librados: starting msgr at 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 1 librados: starting objecter 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 1 Processor -- start 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 1 -- start start 2024-03-29T10:47:13.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 1 librados: setting wanted keys 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 1 librados: calling monclient init 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.821+0000 7f67a3414640 10 monclient: init 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient(hunting): picked mon.noname-b con 0x7f679c142660 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient(hunting): picked mon.noname-c con 0x7f679c0b0da0 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient(hunting): picked mon.noname-a con 0x7f679c1de850 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c0709f0 con 0x7f679c1de850 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c071fa0 con 0x7f679c142660 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c070c80 con 0x7f679c0b0da0 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:13.809 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.822+0000 7f67a3414640 10 monclient(hunting): authenticate will time out at 845.121155s 2024-03-29T10:47:13.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.826+0000 7f67a11b0640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f679c142660 0x7f679c0702e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:59875/0 (socket says 172.21.15.64:59875) 2024-03-29T10:47:13.811 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.826+0000 7f67a11b0640 1 -- 172.21.15.64:0/2445121499 learned_addr learned my addr 172.21.15.64:0/2445121499 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 client.?.objecter ms_handle_connect 0x7f679c1de850 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 client.?.objecter ms_handle_connect 0x7f679c142660 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 client.?.objecter ms_handle_connect 0x7f679c0b0da0 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:13.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318989033 0 0) 0x7f679c0709f0 con 0x7f679c1de850 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.827+0000 7f6791ffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6780004eb0 con 0x7f679c1de850 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894905250 0 0) 0x7f679c070c80 con 0x7f679c0b0da0 2024-03-29T10:47:13.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:47:13.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f679c0709f0 con 0x7f679c0b0da0 2024-03-29T10:47:13.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153111216 0 0) 0x7f679c071fa0 con 0x7f679c142660 2024-03-29T10:47:13.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:47:13.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f679c070c80 con 0x7f679c142660 2024-03-29T10:47:13.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350991591 0 0) 0x7f6780004eb0 con 0x7f679c1de850 2024-03-29T10:47:13.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f679c071fa0 con 0x7f679c1de850 2024-03-29T10:47:13.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f678c0035f0 con 0x7f679c1de850 2024-03-29T10:47:13.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:13.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:13.816 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:13.816 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:13.817 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.817 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.817 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.817 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:13.817 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:13.817 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:13.818 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:13.818 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:13.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:13.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074067738 0 0) 0x7f679c0709f0 con 0x7f679c0b0da0 2024-03-29T10:47:13.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6780006bb0 con 0x7f679c0b0da0 2024-03-29T10:47:13.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6794001b60 con 0x7f679c0b0da0 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.828+0000 7f6791ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.819 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.820 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.820 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:13.820 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:13.820 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:13.820 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:13.820 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:13.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3551150230 0 0) 0x7f679c071fa0 con 0x7f679c1de850 2024-03-29T10:47:13.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 >> v1:172.21.15.64:6790/0 conn(0x7f679c0b0da0 legacy=0x7f679c0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 >> v1:172.21.15.174:6789/0 conn(0x7f679c142660 legacy=0x7f679c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:13.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 1 monclient: found mon.a 2024-03-29T10:47:13.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:13.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:47:13.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c1df3d0 con 0x7f679c1de850 2024-03-29T10:47:13.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:13.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:13.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f6791ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:13.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:47:13.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 10 monclient: _renew_subs 2024-03-29T10:47:13.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:47:13.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c1e7c00 con 0x7f679c1de850 2024-03-29T10:47:13.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:13.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 10 monclient: _renew_subs 2024-03-29T10:47:13.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:47:13.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c1e8130 con 0x7f679c1de850 2024-03-29T10:47:13.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.832+0000 7f67a3414640 1 librados: init done 2024-03-29T10:47:13.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f678c001b10 con 0x7f679c1de850 2024-03-29T10:47:13.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6791ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:13.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f678c0035f0 con 0x7f679c1de850 2024-03-29T10:47:13.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6791ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6791ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:13.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6791ffb640 10 monclient: dump: 2024-03-29T10:47:13.831 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:13.832 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:13.832 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.832 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:13.832 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.832 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:13.832 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:13.833 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:13.833 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:13.833 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:13.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.833+0000 7f6792ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:13.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.835+0000 7f67a3414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.835+0000 7f67a3414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.836+0000 7f67a3414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:47:13.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.836+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c005080 con 0x7f679c1de850 2024-03-29T10:47:13.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.836+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f678c019870 con 0x7f679c1de850 2024-03-29T10:47:13.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.837+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f678c0533a0 con 0x7f679c1de850 2024-03-29T10:47:13.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.837+0000 7f6791ffb640 10 client.4266.objecter ms_dispatch 0x7f679c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:13.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.837+0000 7f6791ffb640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:13.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.837+0000 7f6791ffb640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:13.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.837+0000 7f6791ffb640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T10:47:13.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.838+0000 7f67a11b0640 10 monclient: get_auth_request con 0x7f678006e200 auth_method 0 2024-03-29T10:47:13.835 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.839+0000 7f6791ffb640 10 client.4266.objecter ms_handle_connect 0x7f678006e200 2024-03-29T10:47:13.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.847+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f678c04f640 con 0x7f679c1de850 2024-03-29T10:47:13.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.847+0000 7f6791ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:13.847+0000 7f6791ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:14.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.181+0000 7f67a3414640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:14.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.181+0000 7f67a3414640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:14.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.181+0000 7f67a3414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:47:14.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.181+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f677c005330 con 0x7f679c1de850 2024-03-29T10:47:14.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.192+0000 7f6791ffb640 1 -- 172.21.15.64:0/2445121499 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f678c04f640 con 0x7f679c1de850 2024-03-29T10:47:14.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.192+0000 7f6791ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:14.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.192+0000 7f6791ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:14.174 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:14.175 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:14.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 10 librados: watch_flush enter 2024-03-29T10:47:14.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 10 librados: watch_flush exit 2024-03-29T10:47:14.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T10:47:14.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 10 client.4266.objecter successfully canceled tick 2024-03-29T10:47:14.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 >> v1:172.21.15.174:6816/2473791532 conn(0x7f678006e200 legacy=0x7f67800706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 10 monclient: shutdown 2024-03-29T10:47:14.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:14.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.196+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 >> v1:172.21.15.64:6789/0 conn(0x7f679c1de850 legacy=0x7f679c1decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.198+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 shutdown_connections 2024-03-29T10:47:14.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.198+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 >> 172.21.15.64:0/2445121499 conn(0x7f679c066a00 msgr2=0x7f679c0b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:14.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.200+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 shutdown_connections 2024-03-29T10:47:14.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.200+0000 7f67a3414640 1 -- 172.21.15.64:0/2445121499 wait complete. 2024-03-29T10:47:14.183 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:14.201+0000 7f67a3414640 1 librados: shutdown 2024-03-29T10:47:14.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:21.010 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:21.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.183+0000 7fe2059c1640 10 monclient: get_monmap_and_config 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.183+0000 7fe2059c1640 10 monclient: build_initial_monmap 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.183+0000 7fe2059c1640 10 monclient: monmap: 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:21.184483+0000 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:21.184483+0000 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:21.166 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:21.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.185+0000 7fe2059c1640 1 Processor -- start 2024-03-29T10:47:21.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.185+0000 7fe2059c1640 1 -- start start 2024-03-29T10:47:21.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.185+0000 7fe2059c1640 10 monclient: init 2024-03-29T10:47:21.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:21.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:21.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe2001457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe20013f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe2000a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2000aaef0 con 0x7fe2000a9d70 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe200145bd0 con 0x7fe20013f650 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe200145eb0 con 0x7fe2001457f0 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe2059c1640 10 monclient(hunting): authenticate will time out at 852.485162s 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe1fe7fc640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe20013f650 0x7fe2000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:48986/0 (socket says 172.21.15.64:48986) 2024-03-29T10:47:21.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.186+0000 7fe1fe7fc640 1 -- 172.21.15.64:0/16560914 learned_addr learned my addr 172.21.15.64:0/16560914 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:21.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664339779 0 0) 0x7fe200145eb0 con 0x7fe2001457f0 2024-03-29T10:47:21.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:21.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e8003520 con 0x7fe2001457f0 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315474719 0 0) 0x7fe200145bd0 con 0x7fe20013f650 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e8003840 con 0x7fe20013f650 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162784954 0 0) 0x7fe1e8003520 con 0x7fe2001457f0 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 >> v1:172.21.15.174:6789/0 conn(0x7fe20013f650 legacy=0x7fe2000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 >> v1:172.21.15.64:6789/0 conn(0x7fe2000a9d70 legacy=0x7fe20014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 monclient: found mon.noname-b 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe200146340 con 0x7fe2001457f0 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1f0001670 con 0x7fe2001457f0 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe2059c1640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe2059c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe1fdffb640 10 monclient: dump: 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:21.173 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.187+0000 7fe2059c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f00017d0 con 0x7fe2001457f0 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe1fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe1fdffb640 1 -- 172.21.15.64:0/16560914 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1f0001be0 con 0x7fe2001457f0 2024-03-29T10:47:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 4 set_mon_vals no callback set 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe1fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe1fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe1fdffb640 10 monclient: dump: 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 10 monclient: shutdown 2024-03-29T10:47:21.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:21.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 1 -- 172.21.15.64:0/16560914 >> v1:172.21.15.64:6790/0 conn(0x7fe2001457f0 legacy=0x7fe20013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 1 -- 172.21.15.64:0/16560914 shutdown_connections 2024-03-29T10:47:21.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.188+0000 7fe2059c1640 1 -- 172.21.15.64:0/16560914 >> 172.21.15.64:0/16560914 conn(0x7fe200066a00 msgr2=0x7fe200066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:21.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.189+0000 7fe2059c1640 1 -- 172.21.15.64:0/16560914 shutdown_connections 2024-03-29T10:47:21.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.189+0000 7fe2059c1640 1 -- 172.21.15.64:0/16560914 wait complete. 2024-03-29T10:47:21.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.190+0000 7fe2059c1640 10 monclient: build_initial_monmap 2024-03-29T10:47:21.187 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.190+0000 7fe2059c1640 10 monclient: monmap: 2024-03-29T10:47:21.187 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:21.187 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:21.187 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:21.187 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.190+0000 7fe2059c1640 1 librados: starting msgr at 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.190+0000 7fe2059c1640 1 librados: starting objecter 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.190+0000 7fe2059c1640 1 Processor -- start 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.191+0000 7fe2059c1640 1 -- start start 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.191+0000 7fe2059c1640 1 librados: setting wanted keys 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.191+0000 7fe2059c1640 1 librados: calling monclient init 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.191+0000 7fe2059c1640 10 monclient: init 2024-03-29T10:47:21.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe2059c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe2059c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe2059c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe2000a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe1feffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe2000a9d70 0x7fe2001dbff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:49440/0 (socket says 172.21.15.64:49440) 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe1feffd640 1 -- 172.21.15.64:0/435425968 learned_addr learned my addr 172.21.15.64:0/435425968 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe2059c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe20013f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.192+0000 7fe1e7fff640 10 client.?.objecter ms_handle_connect 0x7fe2000a9d70 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe2059c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe2001457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2001dbb60 con 0x7fe2001457f0 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2001db830 con 0x7fe2000a9d70 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe1e7fff640 10 client.?.objecter ms_handle_connect 0x7fe20013f650 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:21.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2001dc700 con 0x7fe20013f650 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe2059c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.193+0000 7fe2059c1640 10 monclient(hunting): authenticate will time out at 852.492844s 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040961499 0 0) 0x7fe2001db830 con 0x7fe2000a9d70 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec004eb0 con 0x7fe2000a9d70 2024-03-29T10:47:21.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 client.?.objecter ms_handle_connect 0x7fe2001457f0 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142279432 0 0) 0x7fe1ec004eb0 con 0x7fe2000a9d70 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2001db830 con 0x7fe2000a9d70 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1f0001630 con 0x7fe2000a9d70 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): dump: 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:21.191 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569215924 0 0) 0x7fe2001dc700 con 0x7fe20013f650 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:21.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.194+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec004eb0 con 0x7fe20013f650 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989856095 0 0) 0x7fe2001dbb60 con 0x7fe2001457f0 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2001dc700 con 0x7fe2001457f0 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1089552534 0 0) 0x7fe2001db830 con 0x7fe2000a9d70 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 >> v1:172.21.15.64:6790/0 conn(0x7fe20013f650 legacy=0x7fe200137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 >> v1:172.21.15.64:6789/0 conn(0x7fe2001457f0 legacy=0x7fe2001e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 monclient: found mon.b 2024-03-29T10:47:21.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20013bc30 con 0x7fe2000a9d70 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe1e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 5 monclient: authenticate success, global_id 4276 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 10 monclient: _renew_subs 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20006e100 con 0x7fe2000a9d70 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 10 monclient: _renew_subs 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.195+0000 7fe2059c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.196+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe200138de0 con 0x7fe2000a9d70 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.196+0000 7fe2059c1640 1 librados: init done 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f00038a0 con 0x7fe2000a9d70 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1f0003c70 con 0x7fe2000a9d70 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1e7fff640 10 monclient: dump: 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:21.195 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe2059c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.197+0000 7fe1fcff9640 4 set_mon_vals no callback set 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.198+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe1f0003e90 con 0x7fe2000a9d70 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe2059c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe1f0084ba0 con 0x7fe2000a9d70 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1e7fff640 10 client.4276.objecter ms_dispatch 0x7fe200000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1e7fff640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1e7fff640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1e7fff640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T10:47:21.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe2059c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:21.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20013bdc0 con 0x7fe2000a9d70 2024-03-29T10:47:21.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1fe7fc640 10 monclient: get_auth_request con 0x7fe1ec06de20 auth_method 0 2024-03-29T10:47:21.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.199+0000 7fe1e7fff640 10 client.4276.objecter ms_handle_connect 0x7fe1ec06de20 2024-03-29T10:47:21.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.206+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1f0050480 con 0x7fe2000a9d70 2024-03-29T10:47:21.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.206+0000 7fe1e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:21.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.206+0000 7fe1e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:21.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.649+0000 7fe2059c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:21.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.649+0000 7fe2059c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:21.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.649+0000 7fe2059c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:21.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.649+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe200075f10 con 0x7fe2000a9d70 2024-03-29T10:47:21.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.650+0000 7fe1e7fff640 1 -- 172.21.15.64:0/435425968 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe1f00523b0 con 0x7fe2000a9d70 2024-03-29T10:47:21.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.650+0000 7fe1e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:21.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.650+0000 7fe1e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:21.634 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:21.635 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:21.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 10 librados: watch_flush enter 2024-03-29T10:47:21.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 10 librados: watch_flush exit 2024-03-29T10:47:21.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T10:47:21.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 10 client.4276.objecter successfully canceled tick 2024-03-29T10:47:21.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe1ec06de20 legacy=0x7fe1ec0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 10 monclient: shutdown 2024-03-29T10:47:21.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:21.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 >> v1:172.21.15.174:6789/0 conn(0x7fe2000a9d70 legacy=0x7fe2001dbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:21.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 shutdown_connections 2024-03-29T10:47:21.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.654+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 >> 172.21.15.64:0/435425968 conn(0x7fe200066a00 msgr2=0x7fe2000b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:21.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.655+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 shutdown_connections 2024-03-29T10:47:21.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.655+0000 7fe2059c1640 1 -- 172.21.15.64:0/435425968 wait complete. 2024-03-29T10:47:21.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:21.655+0000 7fe2059c1640 1 librados: shutdown 2024-03-29T10:47:21.655 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:28.463 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:28.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.611+0000 7f64670f1640 10 monclient: get_monmap_and_config 2024-03-29T10:47:28.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.611+0000 7f64670f1640 10 monclient: build_initial_monmap 2024-03-29T10:47:28.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.611+0000 7f64670f1640 10 monclient: monmap: 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:28.612512+0000 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:28.612512+0000 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.595 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:28.596 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:28.596 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:28.596 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.612+0000 7f64670f1640 1 Processor -- start 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.612+0000 7f64670f1640 1 -- start start 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.612+0000 7f64670f1640 10 monclient: init 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.613+0000 7f64670f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.613+0000 7f64670f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 10 monclient(hunting): picked mon.noname-a con 0x7f64601457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 10 monclient(hunting): picked mon.noname-b con 0x7f646013f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 10 monclient(hunting): picked mon.noname-c con 0x7f64600a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64600aaef0 con 0x7f64601457f0 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460145bd0 con 0x7f64600a9d70 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460145eb0 con 0x7f646013f650 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f64670f1640 10 monclient(hunting): authenticate will time out at 859.913572s 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f645ffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f646013f650 0x7f64600b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49952/0 (socket says 172.21.15.64:49952) 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.614+0000 7f645ffff640 1 -- 172.21.15.64:0/2764625549 learned_addr learned my addr 172.21.15.64:0/2764625549 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328079293 0 0) 0x7f6460145eb0 con 0x7f646013f650 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6450003520 con 0x7f646013f650 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211474674 0 0) 0x7f6460145bd0 con 0x7f64600a9d70 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:28.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:28.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:47:28.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6460145eb0 con 0x7f64600a9d70 2024-03-29T10:47:28.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4137592305 0 0) 0x7f6450003520 con 0x7f646013f650 2024-03-29T10:47:28.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 >> v1:172.21.15.174:6789/0 conn(0x7f64600a9d70 legacy=0x7f646014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.615+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 >> v1:172.21.15.64:6789/0 conn(0x7f64601457f0 legacy=0x7f646013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460146340 con 0x7f646013f650 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f644c001690 con 0x7f646013f650 2024-03-29T10:47:28.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: dump: 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:28.608 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f64670f1640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f64670f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f644c001870 con 0x7f646013f650 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 1 -- 172.21.15.64:0/2764625549 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f644c001c80 con 0x7f646013f650 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:28.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f645f7fe640 10 monclient: dump: 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:28.610 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f64670f1640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.617+0000 7f64670f1640 4 set_mon_vals no callback set 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.618+0000 7f64670f1640 10 monclient: shutdown 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.618+0000 7f64670f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.618+0000 7f64670f1640 1 -- 172.21.15.64:0/2764625549 >> v1:172.21.15.64:6790/0 conn(0x7f646013f650 legacy=0x7f64600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.618+0000 7f64670f1640 1 -- 172.21.15.64:0/2764625549 shutdown_connections 2024-03-29T10:47:28.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.618+0000 7f64670f1640 1 -- 172.21.15.64:0/2764625549 >> 172.21.15.64:0/2764625549 conn(0x7f6460066a00 msgr2=0x7f6460066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:28.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.619+0000 7f64670f1640 1 -- 172.21.15.64:0/2764625549 shutdown_connections 2024-03-29T10:47:28.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.619+0000 7f64670f1640 1 -- 172.21.15.64:0/2764625549 wait complete. 2024-03-29T10:47:28.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.620+0000 7f64670f1640 10 monclient: build_initial_monmap 2024-03-29T10:47:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.620+0000 7f64670f1640 10 monclient: monmap: 2024-03-29T10:47:28.613 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:28.613 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:28.613 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:28.614 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:28.614 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:28.614 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.614 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 1 librados: starting msgr at 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 1 librados: starting objecter 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 1 Processor -- start 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 1 -- start start 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 1 librados: setting wanted keys 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 1 librados: calling monclient init 2024-03-29T10:47:28.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.621+0000 7f64670f1640 10 monclient: init 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient(hunting): picked mon.noname-c con 0x7f64600a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient(hunting): picked mon.noname-b con 0x7f646013f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient(hunting): picked mon.noname-a con 0x7f64601457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460137810 con 0x7f64601457f0 2024-03-29T10:47:28.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646006ead0 con 0x7f646013f650 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f645ffff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f646013f650 0x7f6460142510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:46736/0 (socket says 172.21.15.64:46736) 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f645ffff640 1 -- 172.21.15.64:0/2554549873 learned_addr learned my addr 172.21.15.64:0/2554549873 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460142c20 con 0x7f64600a9d70 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.622+0000 7f64670f1640 10 monclient(hunting): authenticate will time out at 859.921808s 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 client.?.objecter ms_handle_connect 0x7f646013f650 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 client.?.objecter ms_handle_connect 0x7f64600a9d70 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418286978 0 0) 0x7f646006ead0 con 0x7f646013f650 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:28.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6444004eb0 con 0x7f646013f650 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599117547 0 0) 0x7f6460142c20 con 0x7f64600a9d70 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646006ead0 con 0x7f64600a9d70 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956223020 0 0) 0x7f6444004eb0 con 0x7f646013f650 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6460142c20 con 0x7f646013f650 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f644c001c40 con 0x7f646013f650 2024-03-29T10:47:28.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.623+0000 7f645d7fa640 10 monclient(hunting): dump: 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518473836 0 0) 0x7f646006ead0 con 0x7f64600a9d70 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6444004eb0 con 0x7f64600a9d70 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6454001710 con 0x7f64600a9d70 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 10 monclient(hunting): dump: 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.628 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2514876411 0 0) 0x7f6460142c20 con 0x7f646013f650 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 >> v1:172.21.15.64:6790/0 conn(0x7f64600a9d70 legacy=0x7f64601447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.624+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 >> v1:172.21.15.64:6789/0 conn(0x7f64601457f0 legacy=0x7f64601e64c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 1 monclient: found mon.b 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646006ed40 con 0x7f646013f650 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:28.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.625+0000 7f645d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 10 monclient: _renew_subs 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646006f020 con 0x7f646013f650 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f644c004bf0 con 0x7f646013f650 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f644c004f00 con 0x7f646013f650 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:28.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645d7fa640 10 monclient: dump: 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f645e7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:28.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:28.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 10 monclient: _renew_subs 2024-03-29T10:47:28.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:28.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f646006de90 con 0x7f646013f650 2024-03-29T10:47:28.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.626+0000 7f64670f1640 1 librados: init done 2024-03-29T10:47:28.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.628+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f644c01aed0 con 0x7f646013f650 2024-03-29T10:47:28.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.628+0000 7f64670f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:28.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f64670f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:28.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f64670f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:28.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6460141c00 con 0x7f646013f650 2024-03-29T10:47:28.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f644c084cb0 con 0x7f646013f650 2024-03-29T10:47:28.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f645d7fa640 10 client.4282.objecter ms_dispatch 0x7f6460000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:28.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f645d7fa640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:28.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f645d7fa640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:28.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f645d7fa640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T10:47:28.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f6464e8d640 10 monclient: get_auth_request con 0x7f644406ddb0 auth_method 0 2024-03-29T10:47:28.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.629+0000 7f645d7fa640 10 client.4282.objecter ms_handle_connect 0x7f644406ddb0 2024-03-29T10:47:28.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.638+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6460141c00 con 0x7f646013f650 2024-03-29T10:47:28.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.638+0000 7f645d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:28.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.638+0000 7f645d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:28.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.965+0000 7f64670f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.965+0000 7f64670f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.965+0000 7f64670f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:28.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.965+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6460137630 con 0x7f646013f650 2024-03-29T10:47:28.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.966+0000 7f645d7fa640 1 -- 172.21.15.64:0/2554549873 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f644c050590 con 0x7f646013f650 2024-03-29T10:47:28.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.966+0000 7f645d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.966+0000 7f645d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:28.949 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:28.949 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 10 librados: watch_flush enter 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 10 librados: watch_flush exit 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 10 client.4282.objecter successfully canceled tick 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 >> v1:172.21.15.174:6816/2473791532 conn(0x7f644406ddb0 legacy=0x7f6444070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 10 monclient: shutdown 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 >> v1:172.21.15.174:6789/0 conn(0x7f646013f650 legacy=0x7f6460142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 shutdown_connections 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.969+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 >> 172.21.15.64:0/2554549873 conn(0x7f6460066a00 msgr2=0x7f646013aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.970+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 shutdown_connections 2024-03-29T10:47:28.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.970+0000 7f64670f1640 1 -- 172.21.15.64:0/2554549873 wait complete. 2024-03-29T10:47:28.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:28.970+0000 7f64670f1640 1 librados: shutdown 2024-03-29T10:47:28.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:35.775 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:35.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.941+0000 7fb8ee965640 10 monclient: get_monmap_and_config 2024-03-29T10:47:35.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.941+0000 7fb8ee965640 10 monclient: build_initial_monmap 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.941+0000 7fb8ee965640 10 monclient: monmap: 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:35.942231+0000 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:35.942231+0000 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:35.924 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.942+0000 7fb8ee965640 1 Processor -- start 2024-03-29T10:47:35.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.942+0000 7fb8ee965640 1 -- start start 2024-03-29T10:47:35.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.942+0000 7fb8ee965640 10 monclient: init 2024-03-29T10:47:35.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.943+0000 7fb8ee965640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:35.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.943+0000 7fb8ee965640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 10 monclient(hunting): picked mon.noname-a con 0x7fb8e81457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 10 monclient(hunting): picked mon.noname-b con 0x7fb8e813f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 10 monclient(hunting): picked mon.noname-c con 0x7fb8e80a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e80aaef0 con 0x7fb8e81457f0 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e8145bd0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e8145eb0 con 0x7fb8e813f650 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:35.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.944+0000 7fb8ee965640 10 monclient(hunting): authenticate will time out at 867.243206s 2024-03-29T10:47:35.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.946+0000 7fb8e7fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb8e81457f0 0x7fb8e813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57121/0 (socket says 172.21.15.64:57121) 2024-03-29T10:47:35.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.946+0000 7fb8e7fff640 1 -- 172.21.15.64:0/364573581 learned_addr learned my addr 172.21.15.64:0/364573581 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:35.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533274941 0 0) 0x7fb8e80aaef0 con 0x7fb8e81457f0 2024-03-29T10:47:35.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d4003520 con 0x7fb8e81457f0 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034145659 0 0) 0x7fb8e8145bd0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e80aaef0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 408915944 0 0) 0x7fb8d4003520 con 0x7fb8e81457f0 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.949+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 >> v1:172.21.15.64:6790/0 conn(0x7fb8e813f650 legacy=0x7fb8e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.950+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 >> v1:172.21.15.174:6789/0 conn(0x7fb8e80a9d70 legacy=0x7fb8e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.950+0000 7fb8e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.950+0000 7fb8e6ffd640 1 monclient: found mon.noname-a 2024-03-29T10:47:35.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.950+0000 7fb8e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8146340 con 0x7fb8e81457f0 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb8d0001690 con 0x7fb8e81457f0 2024-03-29T10:47:35.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8e6ffd640 10 monclient: dump: 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:35.938 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8ee965640 5 monclient: authenticate success, global_id 4275 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.951+0000 7fb8ee965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.953+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8d00017f0 con 0x7fb8e81457f0 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.953+0000 7fb8e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8e6ffd640 1 -- 172.21.15.64:0/364573581 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb8d0001c40 con 0x7fb8e81457f0 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8e6ffd640 10 monclient: dump: 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.939 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8ee965640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8ee965640 4 set_mon_vals no callback set 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8ee965640 10 monclient: shutdown 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8ee965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:35.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.954+0000 7fb8ee965640 1 -- 172.21.15.64:0/364573581 >> v1:172.21.15.64:6789/0 conn(0x7fb8e81457f0 legacy=0x7fb8e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.955+0000 7fb8ee965640 1 -- 172.21.15.64:0/364573581 shutdown_connections 2024-03-29T10:47:35.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.955+0000 7fb8ee965640 1 -- 172.21.15.64:0/364573581 >> 172.21.15.64:0/364573581 conn(0x7fb8e8066a00 msgr2=0x7fb8e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.956+0000 7fb8ee965640 1 -- 172.21.15.64:0/364573581 shutdown_connections 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.956+0000 7fb8ee965640 1 -- 172.21.15.64:0/364573581 wait complete. 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.957+0000 7fb8ee965640 10 monclient: build_initial_monmap 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.957+0000 7fb8ee965640 10 monclient: monmap: 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:35.942 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 1 librados: starting msgr at 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 1 librados: starting objecter 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 1 Processor -- start 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 1 -- start start 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 1 librados: setting wanted keys 2024-03-29T10:47:35.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 1 librados: calling monclient init 2024-03-29T10:47:35.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.958+0000 7fb8ee965640 10 monclient: init 2024-03-29T10:47:35.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.959+0000 7fb8ee965640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:35.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.959+0000 7fb8ee965640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:35.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 10 monclient(hunting): picked mon.noname-c con 0x7fb8e80a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8e7fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb8e80a9d70 0x7fb8e81d74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34549/0 (socket says 172.21.15.64:34549) 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8e7fff640 1 -- 172.21.15.64:0/1340226085 learned_addr learned my addr 172.21.15.64:0/1340226085 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8e4ff9640 10 client.?.objecter ms_handle_connect 0x7fb8e80a9d70 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 10 monclient(hunting): picked mon.noname-a con 0x7fb8e813f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 10 monclient(hunting): picked mon.noname-b con 0x7fb8e81457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e81db070 con 0x7fb8e813f650 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e81d7bf0 con 0x7fb8e81457f0 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e81db590 con 0x7fb8e80a9d70 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.963+0000 7fb8ee965640 10 monclient(hunting): authenticate will time out at 867.262799s 2024-03-29T10:47:35.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 10 client.?.objecter ms_handle_connect 0x7fb8e81457f0 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257025450 0 0) 0x7fb8e81d7bf0 con 0x7fb8e81457f0 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.964+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c8004eb0 con 0x7fb8e81457f0 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.965+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731562084 0 0) 0x7fb8e81db590 con 0x7fb8e80a9d70 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.965+0000 7fb8e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.965+0000 7fb8e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.965+0000 7fb8e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.965+0000 7fb8e4ff9640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:47:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.965+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e81d7bf0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 client.?.objecter ms_handle_connect 0x7fb8e813f650 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256238880 0 0) 0x7fb8c8004eb0 con 0x7fb8e81457f0 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e81db590 con 0x7fb8e81457f0 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951379865 0 0) 0x7fb8e81d7bf0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c8004eb0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb8e0001cb0 con 0x7fb8e81457f0 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:35.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb8d0001b00 con 0x7fb8e80a9d70 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510342736 0 0) 0x7fb8e81db070 con 0x7fb8e813f650 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e81d7bf0 con 0x7fb8e813f650 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 938126062 0 0) 0x7fb8c8004eb0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 >> v1:172.21.15.174:6789/0 conn(0x7fb8e81457f0 legacy=0x7fb8e81e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 >> v1:172.21.15.64:6789/0 conn(0x7fb8e813f650 legacy=0x7fb8e8139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 1 monclient: found mon.c 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.967+0000 7fb8e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:35.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e81db720 con 0x7fb8e80a9d70 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 10 monclient: _renew_subs 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e80b0c60 con 0x7fb8e80a9d70 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 10 monclient: _renew_subs 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e81421b0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8ee965640 1 librados: init done 2024-03-29T10:47:35.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8d0001d60 con 0x7fb8e80a9d70 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb8d0004c30 con 0x7fb8e80a9d70 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e4ff9640 10 monclient: dump: 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:35.956 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:35.957 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:35.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.968+0000 7fb8e5ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.969+0000 7fb8ee965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.969+0000 7fb8ee965640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.969+0000 7fb8ee965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.969+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8cc005080 con 0x7fb8e80a9d70 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.972+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fb8d001ac00 con 0x7fb8e80a9d70 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.973+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fb8d00849f0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.973+0000 7fb8e4ff9640 10 client.4286.objecter ms_dispatch 0x7fb8e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.973+0000 7fb8e4ff9640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.973+0000 7fb8e4ff9640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.973+0000 7fb8e4ff9640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.973+0000 7fb8e77fe640 10 monclient: get_auth_request con 0x7fb8c806e020 auth_method 0 2024-03-29T10:47:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.975+0000 7fb8e4ff9640 10 client.4286.objecter ms_handle_connect 0x7fb8c806e020 2024-03-29T10:47:35.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.979+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8d00502d0 con 0x7fb8e80a9d70 2024-03-29T10:47:35.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.979+0000 7fb8e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:35.979+0000 7fb8e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8ee965640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8ee965640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8ee965640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:36.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8cc005330 con 0x7fb8e80a9d70 2024-03-29T10:47:36.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8e4ff9640 1 -- 172.21.15.64:0/1340226085 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fb8d0052200 con 0x7fb8e80a9d70 2024-03-29T10:47:36.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:36.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.295+0000 7fb8e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:36.280 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:36.280 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:36.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.299+0000 7fb8ee965640 10 librados: watch_flush enter 2024-03-29T10:47:36.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.299+0000 7fb8ee965640 10 librados: watch_flush exit 2024-03-29T10:47:36.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.299+0000 7fb8ee965640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T10:47:36.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.299+0000 7fb8ee965640 10 client.4286.objecter successfully canceled tick 2024-03-29T10:47:36.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 >> v1:172.21.15.174:6816/2473791532 conn(0x7fb8c806e020 legacy=0x7fb8c80704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:36.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 10 monclient: shutdown 2024-03-29T10:47:36.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:36.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 >> v1:172.21.15.64:6790/0 conn(0x7fb8e80a9d70 legacy=0x7fb8e81d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:36.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 shutdown_connections 2024-03-29T10:47:36.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 >> 172.21.15.64:0/1340226085 conn(0x7fb8e8066a00 msgr2=0x7fb8e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:36.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 shutdown_connections 2024-03-29T10:47:36.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 -- 172.21.15.64:0/1340226085 wait complete. 2024-03-29T10:47:36.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:36.300+0000 7fb8ee965640 1 librados: shutdown 2024-03-29T10:47:36.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:43.107 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:43.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.279+0000 7fb50e39f640 10 monclient: get_monmap_and_config 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.279+0000 7fb50e39f640 10 monclient: build_initial_monmap 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.280+0000 7fb50e39f640 10 monclient: monmap: 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:43.281819+0000 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:43.281819+0000 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:43.264 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:43.265 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:43.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.281+0000 7fb50e39f640 1 Processor -- start 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.281+0000 7fb50e39f640 1 -- start start 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.281+0000 7fb50e39f640 10 monclient: init 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient(hunting): picked mon.noname-a con 0x7fb5080a9750 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient(hunting): picked mon.noname-c con 0x7fb5080adca0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient(hunting): picked mon.noname-b con 0x7fb508146a80 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080abdc0 con 0x7fb5080a9750 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080b0070 con 0x7fb5080adca0 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080af880 con 0x7fb508146a80 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:43.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.282+0000 7fb50e39f640 10 monclient(hunting): authenticate will time out at 874.582708s 2024-03-29T10:47:43.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.286+0000 7fb50d39d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb5080a9750 0x7fb508143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51755/0 (socket says 172.21.15.64:51755) 2024-03-29T10:47:43.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.286+0000 7fb50d39d640 1 -- 172.21.15.64:0/2138834643 learned_addr learned my addr 172.21.15.64:0/2138834643 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:43.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.289+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001705198 0 0) 0x7fb5080abdc0 con 0x7fb5080a9750 2024-03-29T10:47:43.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.289+0000 7fb4fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:43.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.289+0000 7fb4fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:43.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.289+0000 7fb4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:43.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.289+0000 7fb4fffff640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:47:43.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.289+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f4003520 con 0x7fb5080a9750 2024-03-29T10:47:43.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978100848 0 0) 0x7fb5080b0070 con 0x7fb5080adca0 2024-03-29T10:47:43.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:43.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:43.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:43.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:47:43.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5080abdc0 con 0x7fb5080adca0 2024-03-29T10:47:43.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939448720 0 0) 0x7fb5080abdc0 con 0x7fb5080adca0 2024-03-29T10:47:43.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 >> v1:172.21.15.64:6790/0 conn(0x7fb508146a80 legacy=0x7fb508151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 >> v1:172.21.15.64:6789/0 conn(0x7fb5080a9750 legacy=0x7fb508143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 monclient: found mon.noname-c 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.291+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080ac030 con 0x7fb5080adca0 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 10 monclient: _finish_auth 0 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb50e39f640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb50e39f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb5000019f0 con 0x7fb5080adca0 2024-03-29T10:47:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb4fffff640 10 monclient: dump: 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:43.280 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.292+0000 7fb50e39f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.293+0000 7fb4fffff640 1 -- 172.21.15.64:0/2138834643 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb500001c10 con 0x7fb5080adca0 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.293+0000 7fb4fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.293+0000 7fb50e39f640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.293+0000 7fb50e39f640 4 set_mon_vals no callback set 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.294+0000 7fb50e39f640 10 monclient: shutdown 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.294+0000 7fb50e39f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:43.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.294+0000 7fb50e39f640 1 -- 172.21.15.64:0/2138834643 >> v1:172.21.15.174:6789/0 conn(0x7fb5080adca0 legacy=0x7fb508146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.297+0000 7fb50e39f640 1 -- 172.21.15.64:0/2138834643 shutdown_connections 2024-03-29T10:47:43.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.298+0000 7fb50e39f640 1 -- 172.21.15.64:0/2138834643 >> 172.21.15.64:0/2138834643 conn(0x7fb5080673e0 msgr2=0x7fb5080b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:43.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.298+0000 7fb50e39f640 1 -- 172.21.15.64:0/2138834643 shutdown_connections 2024-03-29T10:47:43.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.298+0000 7fb50e39f640 1 -- 172.21.15.64:0/2138834643 wait complete. 2024-03-29T10:47:43.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 10 monclient: build_initial_monmap 2024-03-29T10:47:43.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 10 monclient: monmap: 2024-03-29T10:47:43.287 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:43.287 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 1 librados: starting msgr at 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 1 librados: starting objecter 2024-03-29T10:47:43.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 1 Processor -- start 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 1 -- start start 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 1 librados: setting wanted keys 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 1 librados: calling monclient init 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.299+0000 7fb50e39f640 10 monclient: init 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.300+0000 7fb50e39f640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.300+0000 7fb50e39f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.300+0000 7fb50e39f640 10 monclient(hunting): picked mon.noname-a con 0x7fb5080a9750 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.300+0000 7fb50e39f640 10 monclient(hunting): picked mon.noname-b con 0x7fb5080adca0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.300+0000 7fb50e39f640 10 monclient(hunting): picked mon.noname-c con 0x7fb508146a80 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.300+0000 7fb50e39f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5081d0710 con 0x7fb5080a9750 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.301+0000 7fb50e39f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5081d1660 con 0x7fb5080adca0 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.301+0000 7fb50e39f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5081ce900 con 0x7fb508146a80 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.301+0000 7fb50e39f640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:43.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.301+0000 7fb50e39f640 10 monclient(hunting): authenticate will time out at 874.600877s 2024-03-29T10:47:43.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.304+0000 7fb50cb9c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fb5080adca0 0x7fb508144f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:37167/0 (socket says 172.21.15.64:37167) 2024-03-29T10:47:43.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.304+0000 7fb50cb9c640 1 -- 172.21.15.64:0/2077560211 learned_addr learned my addr 172.21.15.64:0/2077560211 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:43.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.304+0000 7fb4fdffb640 10 client.?.objecter ms_handle_connect 0x7fb5080adca0 2024-03-29T10:47:43.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.304+0000 7fb4fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.305+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389151829 0 0) 0x7fb5081d1660 con 0x7fb5080adca0 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.305+0000 7fb4fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.305+0000 7fb4fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.305+0000 7fb4fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.305+0000 7fb4fdffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.305+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f4003390 con 0x7fb5080adca0 2024-03-29T10:47:43.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.306+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211559644 0 0) 0x7fb4f4003390 con 0x7fb5080adca0 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.306+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5081d1660 con 0x7fb5080adca0 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.306+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb500003c50 con 0x7fb5080adca0 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.306+0000 7fb4fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.307+0000 7fb4fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.307+0000 7fb4fdffb640 10 monclient(hunting): dump: 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:43.294 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.307+0000 7fb4fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.307+0000 7fb4fdffb640 10 client.?.objecter ms_handle_connect 0x7fb5080a9750 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 client.?.objecter ms_handle_connect 0x7fb508146a80 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1765789772 0 0) 0x7fb5081d1660 con 0x7fb5080adca0 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 >> v1:172.21.15.64:6790/0 conn(0x7fb508146a80 legacy=0x7fb5081ee470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 >> v1:172.21.15.64:6789/0 conn(0x7fb5080a9750 legacy=0x7fb508144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 1 monclient: found mon.b 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:43.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5081cf2e0 con 0x7fb5080adca0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.308+0000 7fb4fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 5 monclient: authenticate success, global_id 4294 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 10 monclient: _renew_subs 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5081cf590 con 0x7fb5080adca0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 10 monclient: _renew_subs 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5081d0140 con 0x7fb5080adca0 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb50e39f640 1 librados: init done 2024-03-29T10:47:43.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb500004240 con 0x7fb5080adca0 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb4fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb500004690 con 0x7fb5080adca0 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb4fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb4fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.309+0000 7fb4fdffb640 10 monclient: dump: 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:43.297 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:43.298 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:43.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.310+0000 7fb4ff7fe640 4 set_mon_vals no callback set 2024-03-29T10:47:43.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.311+0000 7fb50e39f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.311+0000 7fb50e39f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.311+0000 7fb50e39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.311+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e8005080 con 0x7fb5080adca0 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.313+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fb500001d80 con 0x7fb5080adca0 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.313+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fb500085170 con 0x7fb5080adca0 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.313+0000 7fb4fdffb640 10 client.4294.objecter ms_dispatch 0x7fb508000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.313+0000 7fb4fdffb640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.313+0000 7fb4fdffb640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.313+0000 7fb4fdffb640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T10:47:43.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.314+0000 7fb50d39d640 10 monclient: get_auth_request con 0x7fb4f406df00 auth_method 0 2024-03-29T10:47:43.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.317+0000 7fb4fdffb640 10 client.4294.objecter ms_handle_connect 0x7fb4f406df00 2024-03-29T10:47:43.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.322+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb500050a50 con 0x7fb5080adca0 2024-03-29T10:47:43.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.322+0000 7fb4fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:43.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.322+0000 7fb4fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:43.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.660+0000 7fb50e39f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:43.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.660+0000 7fb50e39f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:43.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.660+0000 7fb50e39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:43.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.660+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4e8005330 con 0x7fb5080adca0 2024-03-29T10:47:43.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.661+0000 7fb4fdffb640 1 -- 172.21.15.64:0/2077560211 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fb500052980 con 0x7fb5080adca0 2024-03-29T10:47:43.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.661+0000 7fb4fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:43.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.661+0000 7fb4fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:43.647 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:43.647 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:43.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.664+0000 7fb50e39f640 10 librados: watch_flush enter 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 10 librados: watch_flush exit 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 10 client.4294.objecter successfully canceled tick 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 >> v1:172.21.15.174:6816/2473791532 conn(0x7fb4f406df00 legacy=0x7fb4f40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 10 monclient: shutdown 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 >> v1:172.21.15.174:6789/0 conn(0x7fb5080adca0 legacy=0x7fb508144f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 shutdown_connections 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.665+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 >> 172.21.15.64:0/2077560211 conn(0x7fb5080673e0 msgr2=0x7fb5081ccd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.666+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 shutdown_connections 2024-03-29T10:47:43.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.666+0000 7fb50e39f640 1 -- 172.21.15.64:0/2077560211 wait complete. 2024-03-29T10:47:43.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:43.666+0000 7fb50e39f640 1 librados: shutdown 2024-03-29T10:47:43.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:50.475 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:50.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.651+0000 7f78a9f3b640 10 monclient: get_monmap_and_config 2024-03-29T10:47:50.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.651+0000 7f78a9f3b640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.651+0000 7f78a9f3b640 10 monclient: monmap: 2024-03-29T10:47:50.634 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:50.634 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:50.653217+0000 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:50.653217+0000 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:50.635 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:50.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.654+0000 7f78a9f3b640 1 Processor -- start 2024-03-29T10:47:50.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.654+0000 7f78a9f3b640 1 -- start start 2024-03-29T10:47:50.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.654+0000 7f78a9f3b640 10 monclient: init 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient(hunting): picked mon.noname-b con 0x7f789c0389e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient(hunting): picked mon.noname-c con 0x7f789c038dc0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient(hunting): picked mon.noname-a con 0x7f789c030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c0c54c0 con 0x7f789c030660 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c036430 con 0x7f789c038dc0 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c039840 con 0x7f789c0389e0 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.655+0000 7f78a9f3b640 10 monclient(hunting): authenticate will time out at 881.955183s 2024-03-29T10:47:50.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.658+0000 7f78a3fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f789c038dc0 0x7f789c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:56573/0 (socket says 172.21.15.64:56573) 2024-03-29T10:47:50.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.658+0000 7f78a3fff640 1 -- 172.21.15.64:0/3694952628 learned_addr learned my addr 172.21.15.64:0/3694952628 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:50.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.660+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219848762 0 0) 0x7f789c036430 con 0x7f789c038dc0 2024-03-29T10:47:50.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.660+0000 7f78a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.660+0000 7f78a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.660+0000 7f78a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.660+0000 7f78a37fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:47:50.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.660+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7894003520 con 0x7f789c038dc0 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3271973822 0 0) 0x7f7894003520 con 0x7f789c038dc0 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 >> v1:172.21.15.64:6790/0 conn(0x7f789c0389e0 legacy=0x7f789c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 >> v1:172.21.15.64:6789/0 conn(0x7f789c030660 legacy=0x7f789c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 1 monclient: found mon.noname-c 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c035690 con 0x7f789c038dc0 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7898001690 con 0x7f789c038dc0 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.661+0000 7f78a37fe640 10 monclient: dump: 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:50.648 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7898001860 con 0x7f789c038dc0 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a37fe640 1 -- 172.21.15.64:0/3694952628 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7898001c70 con 0x7f789c038dc0 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a37fe640 10 monclient: dump: 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:50.649 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a9f3b640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a9f3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.662+0000 7f78a9f3b640 4 set_mon_vals no callback set 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.663+0000 7f78a9f3b640 10 monclient: shutdown 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.663+0000 7f78a9f3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.663+0000 7f78a9f3b640 1 -- 172.21.15.64:0/3694952628 >> v1:172.21.15.174:6789/0 conn(0x7f789c038dc0 legacy=0x7f789c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.663+0000 7f78a9f3b640 1 -- 172.21.15.64:0/3694952628 shutdown_connections 2024-03-29T10:47:50.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.664+0000 7f78a9f3b640 1 -- 172.21.15.64:0/3694952628 >> 172.21.15.64:0/3694952628 conn(0x7f789c005c00 msgr2=0x7f789c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.666+0000 7f78a9f3b640 1 -- 172.21.15.64:0/3694952628 shutdown_connections 2024-03-29T10:47:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.666+0000 7f78a9f3b640 1 -- 172.21.15.64:0/3694952628 wait complete. 2024-03-29T10:47:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.667+0000 7f78a9f3b640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.667+0000 7f78a9f3b640 10 monclient: monmap: 2024-03-29T10:47:50.651 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:50.651 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:50.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.667+0000 7f78a9f3b640 1 librados: starting msgr at 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.667+0000 7f78a9f3b640 1 librados: starting objecter 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.667+0000 7f78a9f3b640 1 Processor -- start 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.668+0000 7f78a9f3b640 1 -- start start 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.668+0000 7f78a9f3b640 1 librados: setting wanted keys 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.668+0000 7f78a9f3b640 1 librados: calling monclient init 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.668+0000 7f78a9f3b640 10 monclient: init 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient(hunting): picked mon.noname-a con 0x7f789c030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient(hunting): picked mon.noname-c con 0x7f789c0389e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient(hunting): picked mon.noname-b con 0x7f789c00aad0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c169b30 con 0x7f789c030660 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c16a6d0 con 0x7f789c00aad0 2024-03-29T10:47:50.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c1697c0 con 0x7f789c0389e0 2024-03-29T10:47:50.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.669+0000 7f78a9f3b640 10 monclient(hunting): authenticate will time out at 881.968971s 2024-03-29T10:47:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.672+0000 7f78a3fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f789c0389e0 0x7f789c00a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38295/0 (socket says 172.21.15.64:38295) 2024-03-29T10:47:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.672+0000 7f78a3fff640 1 -- 172.21.15.64:0/1950045903 learned_addr learned my addr 172.21.15.64:0/1950045903 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.672+0000 7f78a17fa640 10 client.?.objecter ms_handle_connect 0x7f789c0389e0 2024-03-29T10:47:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.672+0000 7f78a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.673+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025551444 0 0) 0x7f789c1697c0 con 0x7f789c0389e0 2024-03-29T10:47:50.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7888004eb0 con 0x7f789c0389e0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 10 client.?.objecter ms_handle_connect 0x7f789c00aad0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.674+0000 7f78a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.677+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785646065 0 0) 0x7f789c16a6d0 con 0x7f789c00aad0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.677+0000 7f78a17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.677+0000 7f78a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.677+0000 7f78a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.677+0000 7f78a17fa640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.677+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f789c1697c0 con 0x7f789c00aad0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3799872264 0 0) 0x7f789c1697c0 con 0x7f789c00aad0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f789c16a6d0 con 0x7f789c00aad0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f78a404fdd0 con 0x7f789c00aad0 2024-03-29T10:47:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 10 monclient(hunting): dump: 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:50.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.678+0000 7f78a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1589250277 0 0) 0x7f789c16a6d0 con 0x7f789c00aad0 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 >> v1:172.21.15.64:6790/0 conn(0x7f789c0389e0 legacy=0x7f789c00a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 >> v1:172.21.15.64:6789/0 conn(0x7f789c030660 legacy=0x7f789c169fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 monclient: found mon.b 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c009820 con 0x7f789c00aad0 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a9f3b640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a9f3b640 10 monclient: _renew_subs 2024-03-29T10:47:50.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a9f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:50.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a9f3b640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c00d2b0 con 0x7f789c00aad0 2024-03-29T10:47:50.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78a404f400 con 0x7f789c00aad0 2024-03-29T10:47:50.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f78a4075f10 con 0x7f789c00aad0 2024-03-29T10:47:50.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a9f3b640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:50.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a17fa640 10 monclient: dump: 2024-03-29T10:47:50.669 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.679+0000 7f78a2ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.681+0000 7f78a9f3b640 10 monclient: _renew_subs 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.681+0000 7f78a9f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:50.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.681+0000 7f78a9f3b640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c00c360 con 0x7f789c00aad0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.681+0000 7f78a9f3b640 1 librados: init done 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.683+0000 7f788effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.683+0000 7f788effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.683+0000 7f788effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.683+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c007b70 con 0x7f789c00aad0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.685+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f78a4067bf0 con 0x7f789c00aad0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.685+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f78a40ce040 con 0x7f789c00aad0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.685+0000 7f78a17fa640 10 client.4300.objecter ms_dispatch 0x7f78a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.685+0000 7f78a17fa640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.685+0000 7f78a17fa640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.686+0000 7f78a17fa640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.686+0000 7f78a8f39640 10 monclient: get_auth_request con 0x7f788806db30 auth_method 0 2024-03-29T10:47:50.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.686+0000 7f78a17fa640 10 client.4300.objecter ms_handle_connect 0x7f788806db30 2024-03-29T10:47:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.694+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78a40c9b50 con 0x7f789c00aad0 2024-03-29T10:47:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.694+0000 7f78a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:50.694+0000 7f78a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:51.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.023+0000 7f788effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.023+0000 7f788effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.023+0000 7f788effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:47:51.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.023+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f789c169ce0 con 0x7f789c00aad0 2024-03-29T10:47:51.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.027+0000 7f78a17fa640 1 -- 172.21.15.64:0/1950045903 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f789c169ce0 con 0x7f789c00aad0 2024-03-29T10:47:51.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.027+0000 7f78a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.027+0000 7f78a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:51.011 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:51.011 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:51.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 10 librados: watch_flush enter 2024-03-29T10:47:51.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 10 librados: watch_flush exit 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 10 client.4300.objecter successfully canceled tick 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 >> v1:172.21.15.174:6816/2473791532 conn(0x7f788806db30 legacy=0x7f788806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 10 monclient: shutdown 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 >> v1:172.21.15.174:6789/0 conn(0x7f789c00aad0 legacy=0x7f789c00cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 shutdown_connections 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 >> 172.21.15.64:0/1950045903 conn(0x7f789c005c00 msgr2=0x7f789c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 shutdown_connections 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.033+0000 7f788effd640 1 -- 172.21.15.64:0/1950045903 wait complete. 2024-03-29T10:47:51.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:51.034+0000 7f788effd640 1 librados: shutdown 2024-03-29T10:47:51.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:57.840 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:57.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.008+0000 7fe0c95b2640 10 monclient: get_monmap_and_config 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.008+0000 7fe0c95b2640 10 monclient: build_initial_monmap 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.008+0000 7fe0c95b2640 10 monclient: monmap: 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:47:58.009665+0000 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:47:58.009665+0000 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:47:57.991 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:57.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.010+0000 7fe0c95b2640 1 Processor -- start 2024-03-29T10:47:57.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.010+0000 7fe0c95b2640 1 -- start start 2024-03-29T10:47:57.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.010+0000 7fe0c95b2640 10 monclient: init 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.011+0000 7fe0c95b2640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.011+0000 7fe0c95b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.011+0000 7fe0c95b2640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c40b2800 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c40a9c00 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c40b38c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c4141eb0 con 0x7fe0c40b2800 2024-03-29T10:47:57.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40b22d0 con 0x7fe0c40b38c0 2024-03-29T10:47:57.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c4142d30 con 0x7fe0c40a9c00 2024-03-29T10:47:57.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:57.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.012+0000 7fe0c95b2640 10 monclient(hunting): authenticate will time out at 889.311023s 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.017+0000 7fe0c8db1640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe0c40b38c0 0x7fe0c414d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:58959/0 (socket says 172.21.15.64:58959) 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.017+0000 7fe0c8db1640 1 -- 172.21.15.64:0/3231107296 learned_addr learned my addr 172.21.15.64:0/3231107296 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.017+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235952580 0 0) 0x7fe0c40b22d0 con 0x7fe0c40b38c0 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.017+0000 7fe0c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.017+0000 7fe0c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.017+0000 7fe0c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b0003520 con 0x7fe0c40b38c0 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741445015 0 0) 0x7fe0c4142d30 con 0x7fe0c40a9c00 2024-03-29T10:47:58.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c40b22d0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3211058684 0 0) 0x7fe0b0003520 con 0x7fe0c40b38c0 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 >> v1:172.21.15.64:6790/0 conn(0x7fe0c40a9c00 legacy=0x7fe0c413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 >> v1:172.21.15.64:6789/0 conn(0x7fe0c40b2800 legacy=0x7fe0c4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.018+0000 7fe0c37fe640 1 monclient: found mon.noname-c 2024-03-29T10:47:58.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40aaf90 con 0x7fe0c40b38c0 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe0bc001a10 con 0x7fe0c40b38c0 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c95b2640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c95b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: dump: 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:58.005 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0bc001c30 con 0x7fe0c40b38c0 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c95b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.019+0000 7fe0c37fe640 1 -- 172.21.15.64:0/3231107296 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe0bc001fc0 con 0x7fe0c40b38c0 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.020+0000 7fe0c95b2640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.020+0000 7fe0c95b2640 4 set_mon_vals no callback set 2024-03-29T10:47:58.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 10 monclient: shutdown 2024-03-29T10:47:58.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:58.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 1 -- 172.21.15.64:0/3231107296 >> v1:172.21.15.174:6789/0 conn(0x7fe0c40b38c0 legacy=0x7fe0c414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c37fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:47:58.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 1 -- 172.21.15.64:0/3231107296 shutdown_connections 2024-03-29T10:47:58.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 1 -- 172.21.15.64:0/3231107296 >> 172.21.15.64:0/3231107296 conn(0x7fe0c4066a00 msgr2=0x7fe0c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:58.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 1 -- 172.21.15.64:0/3231107296 shutdown_connections 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.021+0000 7fe0c95b2640 1 -- 172.21.15.64:0/3231107296 wait complete. 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.022+0000 7fe0c95b2640 10 monclient: build_initial_monmap 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.022+0000 7fe0c95b2640 10 monclient: monmap: 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:47:58.009 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 1 librados: starting msgr at 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 1 librados: starting objecter 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 1 Processor -- start 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 1 -- start start 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 1 librados: setting wanted keys 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 1 librados: calling monclient init 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.023+0000 7fe0c95b2640 10 monclient: init 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.024+0000 7fe0c95b2640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.024+0000 7fe0c95b2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c40a9c00 addr v1:172.21.15.64:6790/0 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c40b2800 addr v1:172.21.15.174:6789/0 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c40b38c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:47:58.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c406f790 con 0x7fe0c40b38c0 2024-03-29T10:47:58.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c406fa70 con 0x7fe0c40b2800 2024-03-29T10:47:58.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c4145fc0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:58.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.025+0000 7fe0c95b2640 10 monclient(hunting): authenticate will time out at 889.324328s 2024-03-29T10:47:58.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c3fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe0c40a9c00 0x7fe0c4138080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37945/0 (socket says 172.21.15.64:37945) 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c3fff640 1 -- 172.21.15.64:0/2472967746 learned_addr learned my addr 172.21.15.64:0/2472967746 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 client.?.objecter ms_handle_connect 0x7fe0c40a9c00 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141279261 0 0) 0x7fe0c4145fc0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a4004eb0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.029+0000 7fe0c17fa640 10 client.?.objecter ms_handle_connect 0x7fe0c40b38c0 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745620984 0 0) 0x7fe0c406f790 con 0x7fe0c40b38c0 2024-03-29T10:47:58.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c4145fc0 con 0x7fe0c40b38c0 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150590537 0 0) 0x7fe0a4004eb0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0c406f790 con 0x7fe0c40a9c00 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe0b4001960 con 0x7fe0c40a9c00 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.030+0000 7fe0c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient(hunting): dump: 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:58.016 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327942481 0 0) 0x7fe0c4145fc0 con 0x7fe0c40b38c0 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a4004eb0 con 0x7fe0c40b38c0 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4168785798 0 0) 0x7fe0c406f790 con 0x7fe0c40a9c00 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 >> v1:172.21.15.174:6789/0 conn(0x7fe0c40b2800 legacy=0x7fe0c41de8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 >> v1:172.21.15.64:6789/0 conn(0x7fe0c40b38c0 legacy=0x7fe0c41deff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:58.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 monclient: found mon.c 2024-03-29T10:47:58.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:58.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:58.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c41df700 con 0x7fe0c40a9c00 2024-03-29T10:47:58.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:58.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:58.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.031+0000 7fe0c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:58.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.035+0000 7fe0c95b2640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.035+0000 7fe0c95b2640 10 monclient: _renew_subs 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.035+0000 7fe0c95b2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.035+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c4138790 con 0x7fe0c40a9c00 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.035+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0b4001ac0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c95b2640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c2ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c95b2640 10 monclient: _renew_subs 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c95b2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c406e100 con 0x7fe0c40a9c00 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c95b2640 1 librados: init done 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe0b40038a0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.036+0000 7fe0c17fa640 10 monclient: dump: 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:47:58.021 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:47:58.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.038+0000 7fe0c95b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:58.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.038+0000 7fe0c95b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:58.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.038+0000 7fe0c95b2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:58.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.038+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c4146150 con 0x7fe0c40a9c00 2024-03-29T10:47:58.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.040+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe0b4003a00 con 0x7fe0c40a9c00 2024-03-29T10:47:58.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.041+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe0b4084980 con 0x7fe0c40a9c00 2024-03-29T10:47:58.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.041+0000 7fe0c17fa640 10 client.4304.objecter ms_dispatch 0x7fe0c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:58.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.041+0000 7fe0c17fa640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:58.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.041+0000 7fe0c17fa640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:58.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.041+0000 7fe0c17fa640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:47:58.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.042+0000 7fe0bb7fe640 10 monclient: get_auth_request con 0x7fe0a406de70 auth_method 0 2024-03-29T10:47:58.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.042+0000 7fe0c17fa640 10 client.4304.objecter ms_handle_connect 0x7fe0a406de70 2024-03-29T10:47:58.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.048+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0b4050260 con 0x7fe0c40a9c00 2024-03-29T10:47:58.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.049+0000 7fe0c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:58.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.049+0000 7fe0c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:58.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.384+0000 7fe0c95b2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:58.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.384+0000 7fe0c95b2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:58.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.384+0000 7fe0c95b2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:47:58.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.384+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0c4145db0 con 0x7fe0c40a9c00 2024-03-29T10:47:58.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.386+0000 7fe0c17fa640 1 -- 172.21.15.64:0/2472967746 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe0b4052190 con 0x7fe0c40a9c00 2024-03-29T10:47:58.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.386+0000 7fe0c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:58.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.386+0000 7fe0c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:58.368 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:47:58.368 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.389+0000 7fe0c95b2640 10 librados: watch_flush enter 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.389+0000 7fe0c95b2640 10 librados: watch_flush exit 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.389+0000 7fe0c95b2640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.389+0000 7fe0c95b2640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.390+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe0a406de70 legacy=0x7fe0a4070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.390+0000 7fe0c95b2640 10 monclient: shutdown 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.390+0000 7fe0c95b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:58.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.390+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 >> v1:172.21.15.64:6790/0 conn(0x7fe0c40a9c00 legacy=0x7fe0c4138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.391+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 shutdown_connections 2024-03-29T10:47:58.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.391+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 >> 172.21.15.64:0/2472967746 conn(0x7fe0c4066a00 msgr2=0x7fe0c413bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:58.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.392+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 shutdown_connections 2024-03-29T10:47:58.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.392+0000 7fe0c95b2640 1 -- 172.21.15.64:0/2472967746 wait complete. 2024-03-29T10:47:58.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:47:58.392+0000 7fe0c95b2640 1 librados: shutdown 2024-03-29T10:47:58.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:05.201 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:05.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.374+0000 7fe681c12640 10 monclient: get_monmap_and_config 2024-03-29T10:48:05.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.374+0000 7fe681c12640 10 monclient: build_initial_monmap 2024-03-29T10:48:05.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.374+0000 7fe681c12640 10 monclient: monmap: 2024-03-29T10:48:05.356 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:05.356 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:05.375135+0000 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:05.375135+0000 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:05.357 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:05.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.375+0000 7fe681c12640 1 Processor -- start 2024-03-29T10:48:05.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.375+0000 7fe681c12640 1 -- start start 2024-03-29T10:48:05.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.375+0000 7fe681c12640 10 monclient: init 2024-03-29T10:48:05.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient(hunting): picked mon.noname-b con 0x7fe674038c60 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient(hunting): picked mon.noname-a con 0x7fe6740324b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient(hunting): picked mon.noname-c con 0x7fe6740d11e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740caf80 con 0x7fe6740324b0 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674039850 con 0x7fe6740d11e0 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740c78d0 con 0x7fe674038c60 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:05.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.376+0000 7fe681c12640 10 monclient(hunting): authenticate will time out at 896.675315s 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe681411640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe6740d11e0 0x7fe6740d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:58557/0 (socket says 172.21.15.64:58557) 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe681411640 1 -- 172.21.15.64:0/3531273775 learned_addr learned my addr 172.21.15.64:0/3531273775 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122212463 0 0) 0x7fe674039850 con 0x7fe6740d11e0 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66c003520 con 0x7fe6740d11e0 2024-03-29T10:48:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276976298 0 0) 0x7fe6740c78d0 con 0x7fe674038c60 2024-03-29T10:48:05.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:05.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:05.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:05.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:48:05.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674039850 con 0x7fe674038c60 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3283369573 0 0) 0x7fe66c003520 con 0x7fe6740d11e0 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 >> v1:172.21.15.64:6790/0 conn(0x7fe674038c60 legacy=0x7fe6740c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.380+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 >> v1:172.21.15.64:6789/0 conn(0x7fe6740324b0 legacy=0x7fe6740cb4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 1 monclient: found mon.noname-c 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740c7a60 con 0x7fe6740d11e0 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe668001690 con 0x7fe6740d11e0 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: dump: 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:05.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe681c12640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe681c12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 1 -- 172.21.15.64:0/3531273775 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6680018b0 con 0x7fe6740d11e0 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe67b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe681c12640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.381+0000 7fe681c12640 4 set_mon_vals no callback set 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.382+0000 7fe681c12640 10 monclient: shutdown 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.382+0000 7fe681c12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.382+0000 7fe681c12640 1 -- 172.21.15.64:0/3531273775 >> v1:172.21.15.174:6789/0 conn(0x7fe6740d11e0 legacy=0x7fe6740d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.383+0000 7fe681c12640 1 -- 172.21.15.64:0/3531273775 shutdown_connections 2024-03-29T10:48:05.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.383+0000 7fe681c12640 1 -- 172.21.15.64:0/3531273775 >> 172.21.15.64:0/3531273775 conn(0x7fe674005c20 msgr2=0x7fe674006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:05.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.385+0000 7fe681c12640 1 -- 172.21.15.64:0/3531273775 shutdown_connections 2024-03-29T10:48:05.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.387+0000 7fe681c12640 1 -- 172.21.15.64:0/3531273775 wait complete. 2024-03-29T10:48:05.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.387+0000 7fe681c12640 10 monclient: build_initial_monmap 2024-03-29T10:48:05.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.387+0000 7fe681c12640 10 monclient: monmap: 2024-03-29T10:48:05.370 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 1 librados: starting msgr at 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 1 librados: starting objecter 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 1 Processor -- start 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 1 -- start start 2024-03-29T10:48:05.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 1 librados: setting wanted keys 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 1 librados: calling monclient init 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.388+0000 7fe681c12640 10 monclient: init 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.389+0000 7fe681c12640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.389+0000 7fe681c12640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.389+0000 7fe681c12640 10 monclient(hunting): picked mon.noname-a con 0x7fe674038c60 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 10 monclient(hunting): picked mon.noname-c con 0x7fe6740d11e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 10 monclient(hunting): picked mon.noname-b con 0x7fe674034890 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740353d0 con 0x7fe674038c60 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674036410 con 0x7fe674034890 2024-03-29T10:48:05.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674161190 con 0x7fe6740d11e0 2024-03-29T10:48:05.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:05.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.390+0000 7fe681c12640 10 monclient(hunting): authenticate will time out at 896.688923s 2024-03-29T10:48:05.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.397+0000 7fe67bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe6740d11e0 0x7fe674160370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53897/0 (socket says 172.21.15.64:53897) 2024-03-29T10:48:05.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.397+0000 7fe67bfff640 1 -- 172.21.15.64:0/919254250 learned_addr learned my addr 172.21.15.64:0/919254250 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:05.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.397+0000 7fe6797fa640 10 client.?.objecter ms_handle_connect 0x7fe6740d11e0 2024-03-29T10:48:05.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.397+0000 7fe6797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:05.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789872442 0 0) 0x7fe674161190 con 0x7fe6740d11e0 2024-03-29T10:48:05.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:05.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660004eb0 con 0x7fe6740d11e0 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 client.?.objecter ms_handle_connect 0x7fe674038c60 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604702916 0 0) 0x7fe6740353d0 con 0x7fe674038c60 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.398+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674161190 con 0x7fe674038c60 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394869320 0 0) 0x7fe660004eb0 con 0x7fe6740d11e0 2024-03-29T10:48:05.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6740353d0 con 0x7fe6740d11e0 2024-03-29T10:48:05.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe67c04fdd0 con 0x7fe6740d11e0 2024-03-29T10:48:05.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:05.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:05.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 10 monclient(hunting): dump: 2024-03-29T10:48:05.386 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:05.387 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2887267932 0 0) 0x7fe674161190 con 0x7fe674038c60 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe660004eb0 con 0x7fe674038c60 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2778925570 0 0) 0x7fe6740353d0 con 0x7fe6740d11e0 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 >> v1:172.21.15.174:6789/0 conn(0x7fe674034890 legacy=0x7fe674160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 >> v1:172.21.15.64:6789/0 conn(0x7fe674038c60 legacy=0x7fe6740368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.399+0000 7fe6797fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 1 monclient: found mon.c 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674035600 con 0x7fe6740d11e0 2024-03-29T10:48:05.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe6797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 10 monclient: _renew_subs 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6740358e0 con 0x7fe6740d11e0 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 10 monclient: _renew_subs 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6740c4ab0 con 0x7fe6740d11e0 2024-03-29T10:48:05.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.400+0000 7fe681c12640 1 librados: init done 2024-03-29T10:48:05.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe67c04f400 con 0x7fe6740d11e0 2024-03-29T10:48:05.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe6797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:05.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe67c075f10 con 0x7fe6740d11e0 2024-03-29T10:48:05.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe6797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:05.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe6797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:05.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe6797fa640 10 monclient: dump: 2024-03-29T10:48:05.391 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:05.394 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe681c12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe681c12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe681c12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe674007b90 con 0x7fe6740d11e0 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.404+0000 7fe67a7fc640 4 set_mon_vals no callback set 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.407+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe67c067bf0 con 0x7fe6740d11e0 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe67c0ff430 con 0x7fe6740d11e0 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe6797fa640 10 client.4310.objecter ms_dispatch 0x7fe67c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe6797fa640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe6797fa640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe6797fa640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe680c10640 10 monclient: get_auth_request con 0x7fe66006dc80 auth_method 0 2024-03-29T10:48:05.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.408+0000 7fe6797fa640 10 client.4310.objecter ms_handle_connect 0x7fe66006dc80 2024-03-29T10:48:05.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.415+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe67c0cad70 con 0x7fe6740d11e0 2024-03-29T10:48:05.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.415+0000 7fe6797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:05.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.415+0000 7fe6797fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:05.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.738+0000 7fe681c12640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:05.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.738+0000 7fe681c12640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:05.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.738+0000 7fe681c12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:05.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.738+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6740365c0 con 0x7fe6740d11e0 2024-03-29T10:48:05.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.741+0000 7fe6797fa640 1 -- 172.21.15.64:0/919254250 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe6740365c0 con 0x7fe6740d11e0 2024-03-29T10:48:05.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.741+0000 7fe6797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:05.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.741+0000 7fe6797fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:05.723 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:05.724 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:05.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.744+0000 7fe681c12640 10 librados: watch_flush enter 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 10 librados: watch_flush exit 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 10 client.4310.objecter successfully canceled tick 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe66006dc80 legacy=0x7fe660070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 10 monclient: shutdown 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:05.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.746+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 >> v1:172.21.15.64:6790/0 conn(0x7fe6740d11e0 legacy=0x7fe674160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.747+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 shutdown_connections 2024-03-29T10:48:05.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.747+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 >> 172.21.15.64:0/919254250 conn(0x7fe674005c20 msgr2=0x7fe6740043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:05.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.751+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 shutdown_connections 2024-03-29T10:48:05.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.751+0000 7fe681c12640 1 -- 172.21.15.64:0/919254250 wait complete. 2024-03-29T10:48:05.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:05.751+0000 7fe681c12640 1 librados: shutdown 2024-03-29T10:48:05.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:12.560 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:12.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.724+0000 7fa246894640 10 monclient: get_monmap_and_config 2024-03-29T10:48:12.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.724+0000 7fa246894640 10 monclient: build_initial_monmap 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.724+0000 7fa246894640 10 monclient: monmap: 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:12.725428+0000 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:12.725428+0000 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:12.707 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:12.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.726+0000 7fa246894640 1 Processor -- start 2024-03-29T10:48:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.729+0000 7fa246894640 1 -- start start 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.729+0000 7fa246894640 10 monclient: init 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.729+0000 7fa246894640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.729+0000 7fa246894640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.729+0000 7fa246894640 10 monclient(hunting): picked mon.noname-b con 0x7fa2400b2800 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 10 monclient(hunting): picked mon.noname-c con 0x7fa240142b20 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 10 monclient(hunting): picked mon.noname-a con 0x7fa24013c040 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2400aad10 con 0x7fa24013c040 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa240141f30 con 0x7fa240142b20 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2400aaf60 con 0x7fa2400b2800 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.730+0000 7fa246894640 10 monclient(hunting): authenticate will time out at 904.028921s 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.733+0000 7fa245892640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fa2400b2800 0x7fa240142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41171/0 (socket says 172.21.15.64:41171) 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.733+0000 7fa245892640 1 -- 172.21.15.64:0/586909764 learned_addr learned my addr 172.21.15.64:0/586909764 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465864752 0 0) 0x7fa240141f30 con 0x7fa240142b20 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient(hunting): _negotiate 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa22c003520 con 0x7fa240142b20 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2060420708 0 0) 0x7fa22c003520 con 0x7fa240142b20 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 >> v1:172.21.15.64:6790/0 conn(0x7fa2400b2800 legacy=0x7fa240142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 >> v1:172.21.15.64:6789/0 conn(0x7fa24013c040 legacy=0x7fa24014d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: _finish_hunting 0 2024-03-29T10:48:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 monclient: found mon.noname-c 2024-03-29T10:48:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:48:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2400b3900 con 0x7fa240142b20 2024-03-29T10:48:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: _finish_auth 0 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: _check_auth_tickets 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa2300015c0 con 0x7fa240142b20 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa244890640 10 monclient: dump: 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:12.722 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa246894640 5 monclient: authenticate success, global_id 4309 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.734+0000 7fa246894640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa230001720 con 0x7fa240142b20 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa244890640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa244890640 1 -- 172.21.15.64:0/586909764 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa230001b70 con 0x7fa240142b20 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa244890640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa244890640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa244890640 10 monclient: dump: 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:12.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa246894640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.735+0000 7fa246894640 4 set_mon_vals no callback set 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.736+0000 7fa246894640 10 monclient: shutdown 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.736+0000 7fa246894640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:12.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.736+0000 7fa246894640 1 -- 172.21.15.64:0/586909764 >> v1:172.21.15.174:6789/0 conn(0x7fa240142b20 legacy=0x7fa24013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.740+0000 7fa246894640 1 -- 172.21.15.64:0/586909764 shutdown_connections 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.741+0000 7fa246894640 1 -- 172.21.15.64:0/586909764 >> 172.21.15.64:0/586909764 conn(0x7fa24006d0e0 msgr2=0x7fa240066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.741+0000 7fa246894640 1 -- 172.21.15.64:0/586909764 shutdown_connections 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.741+0000 7fa246894640 1 -- 172.21.15.64:0/586909764 wait complete. 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.742+0000 7fa246894640 10 monclient: build_initial_monmap 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.742+0000 7fa246894640 10 monclient: monmap: 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:12.726 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 1 librados: starting msgr at 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 1 librados: starting objecter 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 1 Processor -- start 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 1 -- start start 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 1 librados: setting wanted keys 2024-03-29T10:48:12.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 1 librados: calling monclient init 2024-03-29T10:48:12.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.743+0000 7fa246894640 10 monclient: init 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient(hunting): picked mon.noname-b con 0x7fa2400b2800 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient(hunting): picked mon.noname-a con 0x7fa24013c040 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient(hunting): picked mon.noname-c con 0x7fa240142b20 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2401d72d0 con 0x7fa24013c040 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2401434b0 con 0x7fa2400b2800 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2401431d0 con 0x7fa240142b20 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:12.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.744+0000 7fa246894640 10 monclient(hunting): authenticate will time out at 904.043702s 2024-03-29T10:48:12.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.750+0000 7fa245091640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa24013c040 0x7fa2401d6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37859/0 (socket says 172.21.15.64:37859) 2024-03-29T10:48:12.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.750+0000 7fa245091640 1 -- 172.21.15.64:0/1178118773 learned_addr learned my addr 172.21.15.64:0/1178118773 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 client.?.objecter ms_handle_connect 0x7fa240142b20 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753137948 0 0) 0x7fa2401431d0 con 0x7fa240142b20 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa224004eb0 con 0x7fa240142b20 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184094969 0 0) 0x7fa224004eb0 con 0x7fa240142b20 2024-03-29T10:48:12.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2401431d0 con 0x7fa240142b20 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa23c0018b0 con 0x7fa240142b20 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): dump: 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:12.737 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.753+0000 7fa2367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4095044064 0 0) 0x7fa2401431d0 con 0x7fa240142b20 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 >> v1:172.21.15.174:6789/0 conn(0x7fa2400b2800 legacy=0x7fa24013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 >> v1:172.21.15.64:6789/0 conn(0x7fa24013c040 legacy=0x7fa2401d6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 monclient: found mon.c 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240144050 con 0x7fa240142b20 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:12.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa23c001a10 con 0x7fa240142b20 2024-03-29T10:48:12.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:12.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2377fe640 4 set_mon_vals no callback set 2024-03-29T10:48:12.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:48:12.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa23c003870 con 0x7fa240142b20 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 10 monclient: _renew_subs 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa240134950 con 0x7fa240142b20 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 10 monclient: _renew_subs 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24013e6f0 con 0x7fa240142b20 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.754+0000 7fa246894640 1 librados: init done 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.755+0000 7fa2367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.755+0000 7fa2367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.755+0000 7fa2367fc640 10 monclient: dump: 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:12.740 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.756+0000 7fa246894640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.756+0000 7fa246894640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.756+0000 7fa246894640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:12.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.756+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24004efb0 con 0x7fa240142b20 2024-03-29T10:48:12.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.758+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fa23c0039d0 con 0x7fa240142b20 2024-03-29T10:48:12.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.759+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fa23c083930 con 0x7fa240142b20 2024-03-29T10:48:12.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.759+0000 7fa2367fc640 10 client.4316.objecter ms_dispatch 0x7fa240000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:12.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.759+0000 7fa2367fc640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:12.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.759+0000 7fa2367fc640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:12.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.759+0000 7fa2367fc640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T10:48:12.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.760+0000 7fa245892640 10 monclient: get_auth_request con 0x7fa22406dc00 auth_method 0 2024-03-29T10:48:12.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.761+0000 7fa2367fc640 10 client.4316.objecter ms_handle_connect 0x7fa22406dc00 2024-03-29T10:48:12.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.765+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa23c04f040 con 0x7fa240142b20 2024-03-29T10:48:12.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.765+0000 7fa2367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:12.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:12.765+0000 7fa2367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:13.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.101+0000 7fa246894640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:13.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.102+0000 7fa246894640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:13.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.102+0000 7fa246894640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:13.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.102+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa24004f730 con 0x7fa240142b20 2024-03-29T10:48:13.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.103+0000 7fa2367fc640 1 -- 172.21.15.64:0/1178118773 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fa23c051140 con 0x7fa240142b20 2024-03-29T10:48:13.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.103+0000 7fa2367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:13.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.103+0000 7fa2367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:13.085 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:13.085 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:13.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.106+0000 7fa246894640 10 librados: watch_flush enter 2024-03-29T10:48:13.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.106+0000 7fa246894640 10 librados: watch_flush exit 2024-03-29T10:48:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.106+0000 7fa246894640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T10:48:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.106+0000 7fa246894640 10 client.4316.objecter successfully canceled tick 2024-03-29T10:48:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.107+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 >> v1:172.21.15.174:6816/2473791532 conn(0x7fa22406dc00 legacy=0x7fa2240700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.107+0000 7fa246894640 10 monclient: shutdown 2024-03-29T10:48:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.107+0000 7fa246894640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.107+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 >> v1:172.21.15.64:6790/0 conn(0x7fa240142b20 legacy=0x7fa2401dd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.110+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 shutdown_connections 2024-03-29T10:48:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.110+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 >> 172.21.15.64:0/1178118773 conn(0x7fa24006d0e0 msgr2=0x7fa2400b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.110+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 shutdown_connections 2024-03-29T10:48:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.110+0000 7fa246894640 1 -- 172.21.15.64:0/1178118773 wait complete. 2024-03-29T10:48:13.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:13.112+0000 7fa246894640 1 librados: shutdown 2024-03-29T10:48:13.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:19.920 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.116+0000 7f4e5480f640 10 monclient: get_monmap_and_config 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.116+0000 7f4e5480f640 10 monclient: build_initial_monmap 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.116+0000 7f4e5480f640 10 monclient: monmap: 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:20.116890+0000 2024-03-29T10:48:20.098 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:20.116890+0000 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.116+0000 7f4e5480f640 1 Processor -- start 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.117+0000 7f4e5480f640 1 -- start start 2024-03-29T10:48:20.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.117+0000 7f4e5480f640 10 monclient: init 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.118+0000 7f4e5480f640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.118+0000 7f4e5480f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.118+0000 7f4e5480f640 10 monclient(hunting): picked mon.noname-c con 0x7f4e501457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.118+0000 7f4e5480f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5013f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e5480f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e500a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e5480f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500aaef0 con 0x7f4e500a9d70 2024-03-29T10:48:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e4e59c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f4e501457f0 0x7f4e5013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:57066/0 (socket says 172.21.15.64:57066) 2024-03-29T10:48:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e4ed9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4e500a9d70 0x7f4e5014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:53550/0 (socket says 172.21.15.64:53550) 2024-03-29T10:48:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e4ed9d640 1 -- 172.21.15.64:0/3129711146 learned_addr learned my addr 172.21.15.64:0/3129711146 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50145bd0 con 0x7f4e501457f0 2024-03-29T10:48:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50145eb0 con 0x7f4e5013f650 2024-03-29T10:48:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e5480f640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.119+0000 7f4e5480f640 10 monclient(hunting): authenticate will time out at 911.418254s 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275379948 0 0) 0x7f4e50145eb0 con 0x7f4e5013f650 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e38003520 con 0x7f4e5013f650 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823951584 0 0) 0x7f4e500aaef0 con 0x7f4e500a9d70 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:48:20.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e50145eb0 con 0x7f4e500a9d70 2024-03-29T10:48:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989415554 0 0) 0x7f4e50145bd0 con 0x7f4e501457f0 2024-03-29T10:48:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.120+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e500aaef0 con 0x7f4e501457f0 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750698349 0 0) 0x7f4e38003520 con 0x7f4e5013f650 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 >> v1:172.21.15.174:6789/0 conn(0x7f4e501457f0 legacy=0x7f4e5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 >> v1:172.21.15.64:6789/0 conn(0x7f4e500a9d70 legacy=0x7f4e5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 monclient: found mon.noname-b 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:48:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e50146340 con 0x7f4e5013f650 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: _finish_auth 0 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e5480f640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e5480f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e40001580 con 0x7f4e5013f650 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: dump: 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.106 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e5480f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e400017d0 con 0x7f4e5013f650 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e5480f640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.121+0000 7f4e5480f640 4 set_mon_vals no callback set 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e4d59a640 1 -- 172.21.15.64:0/3129711146 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e40001b60 con 0x7f4e5013f650 2024-03-29T10:48:20.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e4d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e4d59a640 10 monclient: dump: 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:20.108 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 10 monclient: shutdown 2024-03-29T10:48:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 >> v1:172.21.15.64:6790/0 conn(0x7f4e5013f650 legacy=0x7f4e500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 shutdown_connections 2024-03-29T10:48:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 >> 172.21.15.64:0/3129711146 conn(0x7f4e50066a00 msgr2=0x7f4e50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 shutdown_connections 2024-03-29T10:48:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.122+0000 7f4e5480f640 1 -- 172.21.15.64:0/3129711146 wait complete. 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.123+0000 7f4e5480f640 10 monclient: build_initial_monmap 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.123+0000 7f4e5480f640 10 monclient: monmap: 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 1 librados: starting msgr at 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 1 librados: starting objecter 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 1 Processor -- start 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 1 -- start start 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 1 librados: setting wanted keys 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 1 librados: calling monclient init 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.124+0000 7f4e5480f640 10 monclient: init 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e500a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5013f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient(hunting): picked mon.noname-c con 0x7f4e501457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50140110 con 0x7f4e500a9d70 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e501e52f0 con 0x7f4e5013f650 2024-03-29T10:48:20.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50140f40 con 0x7f4e501457f0 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e5480f640 10 monclient(hunting): authenticate will time out at 911.424347s 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e4e59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4e500a9d70 0x7f4e501e76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:53762/0 (socket says 172.21.15.64:53762) 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e4e59c640 1 -- 172.21.15.64:0/3829935757 learned_addr learned my addr 172.21.15.64:0/3829935757 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e3effd640 10 client.?.objecter ms_handle_connect 0x7f4e500a9d70 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.125+0000 7f4e3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 client.?.objecter ms_handle_connect 0x7f4e5013f650 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 client.?.objecter ms_handle_connect 0x7f4e501457f0 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942451790 0 0) 0x7f4e50140110 con 0x7f4e500a9d70 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:20.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e30004eb0 con 0x7f4e500a9d70 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962739022 0 0) 0x7f4e501e52f0 con 0x7f4e5013f650 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e50140110 con 0x7f4e5013f650 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537666901 0 0) 0x7f4e50140f40 con 0x7f4e501457f0 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e501e52f0 con 0x7f4e501457f0 2024-03-29T10:48:20.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3783673454 0 0) 0x7f4e30004eb0 con 0x7f4e500a9d70 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e50140f40 con 0x7f4e500a9d70 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e440018c0 con 0x7f4e500a9d70 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): dump: 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:20.122 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.126+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2626903759 0 0) 0x7f4e501e52f0 con 0x7f4e501457f0 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e30004eb0 con 0x7f4e501457f0 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e48001bf0 con 0x7f4e501457f0 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient(hunting): dump: 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:20.123 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3498406809 0 0) 0x7f4e50140110 con 0x7f4e5013f650 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e501e52f0 con 0x7f4e5013f650 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e40001ae0 con 0x7f4e5013f650 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient(hunting): dump: 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.124 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3354102496 0 0) 0x7f4e50140f40 con 0x7f4e500a9d70 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 >> v1:172.21.15.64:6790/0 conn(0x7f4e501457f0 legacy=0x7f4e501e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 >> v1:172.21.15.174:6789/0 conn(0x7f4e5013f650 legacy=0x7f4e501e44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 monclient: found mon.a 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:20.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e50141220 con 0x7f4e500a9d70 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient: _finish_auth 0 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 10 monclient: _renew_subs 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5013fda0 con 0x7f4e500a9d70 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 10 monclient: _renew_subs 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e50136800 con 0x7f4e500a9d70 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.127+0000 7f4e5480f640 1 librados: init done 2024-03-29T10:48:20.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e440047e0 con 0x7f4e500a9d70 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e44004af0 con 0x7f4e500a9d70 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e3effd640 10 monclient: dump: 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:20.127 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.128+0000 7f4e4cd99640 4 set_mon_vals no callback set 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.129+0000 7f4e5480f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.131+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f4e4401aac0 con 0x7f4e500a9d70 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.131+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f4e44084a70 con 0x7f4e500a9d70 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e3effd640 10 client.4299.objecter ms_dispatch 0x7f4e50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e3effd640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e3effd640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e3effd640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e5480f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e5480f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:20.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e34005080 con 0x7f4e500a9d70 2024-03-29T10:48:20.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.132+0000 7f4e4dd9b640 10 monclient: get_auth_request con 0x7f4e3006e110 auth_method 0 2024-03-29T10:48:20.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.133+0000 7f4e3effd640 10 client.4299.objecter ms_handle_connect 0x7f4e3006e110 2024-03-29T10:48:20.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.142+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e44050180 con 0x7f4e500a9d70 2024-03-29T10:48:20.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.142+0000 7f4e3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:20.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.142+0000 7f4e3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:20.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.541+0000 7f4e5480f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:20.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.541+0000 7f4e5480f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:20.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.541+0000 7f4e5480f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:20.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.541+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e34005330 con 0x7f4e500a9d70 2024-03-29T10:48:20.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.542+0000 7f4e3effd640 1 -- 172.21.15.64:0/3829935757 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f4e44052280 con 0x7f4e500a9d70 2024-03-29T10:48:20.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.542+0000 7f4e3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:20.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.542+0000 7f4e3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:20.524 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:20.525 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 10 librados: watch_flush enter 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 10 librados: watch_flush exit 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 10 client.4299.objecter successfully canceled tick 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 >> v1:172.21.15.174:6816/2473791532 conn(0x7f4e3006e110 legacy=0x7f4e300705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 10 monclient: shutdown 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 >> v1:172.21.15.64:6789/0 conn(0x7f4e500a9d70 legacy=0x7f4e501e76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 shutdown_connections 2024-03-29T10:48:20.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 >> 172.21.15.64:0/3829935757 conn(0x7f4e50066a00 msgr2=0x7f4e5013b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:20.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 shutdown_connections 2024-03-29T10:48:20.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 -- 172.21.15.64:0/3829935757 wait complete. 2024-03-29T10:48:20.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:20.546+0000 7f4e5480f640 1 librados: shutdown 2024-03-29T10:48:20.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:27.350 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:27.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.514+0000 7f96bcda8640 10 monclient: get_monmap_and_config 2024-03-29T10:48:27.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.514+0000 7f96bcda8640 10 monclient: build_initial_monmap 2024-03-29T10:48:27.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.514+0000 7f96bcda8640 10 monclient: monmap: 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:27.515412+0000 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:27.515412+0000 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:27.497 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.515+0000 7f96bcda8640 1 Processor -- start 2024-03-29T10:48:27.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.515+0000 7f96bcda8640 1 -- start start 2024-03-29T10:48:27.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.515+0000 7f96bcda8640 10 monclient: init 2024-03-29T10:48:27.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:27.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 10 monclient(hunting): picked mon.noname-c con 0x7f96b81457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 10 monclient(hunting): picked mon.noname-b con 0x7f96b813f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 10 monclient(hunting): picked mon.noname-a con 0x7f96b80a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b80aaef0 con 0x7f96b80a9d70 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96b5d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f96b813f650 0x7f96b80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50608/0 (socket says 172.21.15.64:50608) 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96b5d9b640 1 -- 172.21.15.64:0/4257084562 learned_addr learned my addr 172.21.15.64:0/4257084562 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.516+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b8145bd0 con 0x7f96b81457f0 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b8145eb0 con 0x7f96b813f650 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96bcda8640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:27.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96bcda8640 10 monclient(hunting): authenticate will time out at 918.815966s 2024-03-29T10:48:27.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775075027 0 0) 0x7f96b80aaef0 con 0x7f96b80a9d70 2024-03-29T10:48:27.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:27.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:27.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:27.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a0003520 con 0x7f96b80a9d70 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278558646 0 0) 0x7f96b8145eb0 con 0x7f96b813f650 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.517+0000 7f96b559a640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b80aaef0 con 0x7f96b813f650 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667754399 0 0) 0x7f96a0003520 con 0x7f96b80a9d70 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 >> v1:172.21.15.64:6790/0 conn(0x7f96b813f650 legacy=0x7f96b80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 >> v1:172.21.15.174:6789/0 conn(0x7f96b81457f0 legacy=0x7f96b813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 monclient: found mon.noname-a 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:27.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8146340 con 0x7f96b80a9d70 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: _finish_auth 0 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f96b0001a10 con 0x7f96b80a9d70 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: dump: 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:27.502 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96bcda8640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96bcda8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b0001c30 con 0x7f96b80a9d70 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 1 -- 172.21.15.64:0/4257084562 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f96b0001fc0 con 0x7f96b80a9d70 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96b559a640 10 monclient: dump: 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:27.503 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96bcda8640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:27.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.518+0000 7f96bcda8640 4 set_mon_vals no callback set 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.519+0000 7f96bcda8640 10 monclient: shutdown 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.519+0000 7f96bcda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.519+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 >> v1:172.21.15.64:6789/0 conn(0x7f96b80a9d70 legacy=0x7f96b814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.519+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 shutdown_connections 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.519+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 >> 172.21.15.64:0/4257084562 conn(0x7f96b8066a00 msgr2=0x7f96b8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.520+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 shutdown_connections 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.520+0000 7f96bcda8640 1 -- 172.21.15.64:0/4257084562 wait complete. 2024-03-29T10:48:27.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.521+0000 7f96bcda8640 10 monclient: build_initial_monmap 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.521+0000 7f96bcda8640 10 monclient: monmap: 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.521+0000 7f96bcda8640 1 librados: starting msgr at 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.521+0000 7f96bcda8640 1 librados: starting objecter 2024-03-29T10:48:27.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.521+0000 7f96bcda8640 1 Processor -- start 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 1 -- start start 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 1 librados: setting wanted keys 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 1 librados: calling monclient init 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 10 monclient: init 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 10 monclient(hunting): picked mon.noname-c con 0x7f96b80a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.522+0000 7f96bcda8640 10 monclient(hunting): picked mon.noname-a con 0x7f96b813f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96b659c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f96b80a9d70 0x7f96b81da6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51186/0 (socket says 172.21.15.64:51186) 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96b5d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f96b813f650 0x7f96b8139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35160/0 (socket says 172.21.15.64:35160) 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96b659c640 1 -- 172.21.15.64:0/78746786 learned_addr learned my addr 172.21.15.64:0/78746786 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96bcda8640 10 monclient(hunting): picked mon.noname-b con 0x7f96b81457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:27.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b80af3e0 con 0x7f96b813f650 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96a6ffd640 10 client.?.objecter ms_handle_connect 0x7f96b80a9d70 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96a6ffd640 10 client.?.objecter ms_handle_connect 0x7f96b813f650 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b80adcb0 con 0x7f96b81457f0 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b80ae1d0 con 0x7f96b80a9d70 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96bcda8640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96bcda8640 10 monclient(hunting): authenticate will time out at 918.822606s 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96a6ffd640 10 client.?.objecter ms_handle_connect 0x7f96b81457f0 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.523+0000 7f96a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197376670 0 0) 0x7f96b80af3e0 con 0x7f96b813f650 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:27.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698004eb0 con 0x7f96b813f650 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244607047 0 0) 0x7f96b80adcb0 con 0x7f96b81457f0 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b80af3e0 con 0x7f96b81457f0 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237775606 0 0) 0x7f96b80ae1d0 con 0x7f96b80a9d70 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b80adcb0 con 0x7f96b80a9d70 2024-03-29T10:48:27.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478113994 0 0) 0x7f9698004eb0 con 0x7f96b813f650 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96b80ae1d0 con 0x7f96b813f650 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f96ac001930 con 0x7f96b813f650 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:27.513 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.524+0000 7f96a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857851082 0 0) 0x7f96b80af3e0 con 0x7f96b81457f0 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9698004eb0 con 0x7f96b81457f0 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f96b0001950 con 0x7f96b81457f0 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.514 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2584075155 0 0) 0x7f96b80adcb0 con 0x7f96b80a9d70 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96b80af3e0 con 0x7f96b80a9d70 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f96a8001890 con 0x7f96b80a9d70 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:27.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2163721878 0 0) 0x7f96b80ae1d0 con 0x7f96b813f650 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 >> v1:172.21.15.64:6790/0 conn(0x7f96b80a9d70 legacy=0x7f96b81da6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 >> v1:172.21.15.174:6789/0 conn(0x7f96b81457f0 legacy=0x7f96b81e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 monclient: found mon.a 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b80ae360 con 0x7f96b813f650 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 10 monclient: _renew_subs 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b80ad790 con 0x7f96b813f650 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:27.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 10 monclient: _renew_subs 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b81dadd0 con 0x7f96b813f650 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.525+0000 7f96bcda8640 1 librados: init done 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96ac001d40 con 0x7f96b813f650 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f96ac004980 con 0x7f96b813f650 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96b4d99640 4 set_mon_vals no callback set 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.526+0000 7f96a6ffd640 10 monclient: dump: 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.518 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:27.519 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:27.519 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:27.519 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:27.519 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:27.519 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:27.519 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.527+0000 7f96bcda8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.527+0000 7f96bcda8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.527+0000 7f96bcda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.527+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c005080 con 0x7f96b813f650 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.528+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f96ac01a950 con 0x7f96b813f650 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f96ac084980 con 0x7f96b813f650 2024-03-29T10:48:27.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96a6ffd640 10 client.4305.objecter ms_dispatch 0x7f96b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96a6ffd640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96a6ffd640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96a6ffd640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96b659c640 10 monclient: get_auth_request con 0x7f969806e170 auth_method 0 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.529+0000 7f96a6ffd640 10 client.4305.objecter ms_handle_connect 0x7f969806e170 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.537+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96ac0502c0 con 0x7f96b813f650 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.537+0000 7f96a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:27.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.537+0000 7f96a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:27.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.944+0000 7f96bcda8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:27.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.944+0000 7f96bcda8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:27.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.944+0000 7f96bcda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:48:27.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.944+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f969c005330 con 0x7f96b813f650 2024-03-29T10:48:27.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.945+0000 7f96a6ffd640 1 -- 172.21.15.64:0/78746786 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f96ac0521f0 con 0x7f96b813f650 2024-03-29T10:48:27.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.945+0000 7f96a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:27.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.945+0000 7f96a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:27.927 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:27.927 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:27.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.948+0000 7f96bcda8640 10 librados: watch_flush enter 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.948+0000 7f96bcda8640 10 librados: watch_flush exit 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.948+0000 7f96bcda8640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.948+0000 7f96bcda8640 10 client.4305.objecter successfully canceled tick 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 >> v1:172.21.15.174:6816/2473791532 conn(0x7f969806e170 legacy=0x7f9698070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 10 monclient: shutdown 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 >> v1:172.21.15.64:6789/0 conn(0x7f96b813f650 legacy=0x7f96b8139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 shutdown_connections 2024-03-29T10:48:27.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 >> 172.21.15.64:0/78746786 conn(0x7f96b8066a00 msgr2=0x7f96b813ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:27.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 shutdown_connections 2024-03-29T10:48:27.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.949+0000 7f96bcda8640 1 -- 172.21.15.64:0/78746786 wait complete. 2024-03-29T10:48:27.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:27.950+0000 7f96bcda8640 1 librados: shutdown 2024-03-29T10:48:27.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:34.756 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.904+0000 7f13a457b640 10 monclient: get_monmap_and_config 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.904+0000 7f13a457b640 10 monclient: build_initial_monmap 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.904+0000 7f13a457b640 10 monclient: monmap: 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:34.906028+0000 2024-03-29T10:48:34.887 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:34.906028+0000 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.905+0000 7f13a457b640 1 Processor -- start 2024-03-29T10:48:34.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.905+0000 7f13a457b640 1 -- start start 2024-03-29T10:48:34.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.905+0000 7f13a457b640 10 monclient: init 2024-03-29T10:48:34.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient(hunting): picked mon.noname-b con 0x7f139c0b0500 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient(hunting): picked mon.noname-c con 0x7f139c0b08e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient(hunting): picked mon.noname-a con 0x7f139c0aeca0 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c140620 con 0x7f139c0aeca0 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c13daf0 con 0x7f139c0b08e0 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c0ae180 con 0x7f139c0b0500 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a457b640 10 monclient(hunting): authenticate will time out at 926.206524s 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a1b16640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f139c0b08e0 0x7f139c13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:35960/0 (socket says 172.21.15.64:35960) 2024-03-29T10:48:34.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.906+0000 7f13a1b16640 1 -- 172.21.15.64:0/3913067909 learned_addr learned my addr 172.21.15.64:0/3913067909 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:34.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488572507 0 0) 0x7f139c13daf0 con 0x7f139c0b08e0 2024-03-29T10:48:34.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1388003520 con 0x7f139c0b08e0 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802347538 0 0) 0x7f139c0ae180 con 0x7f139c0b0500 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c13daf0 con 0x7f139c0b0500 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 990043429 0 0) 0x7f139c13daf0 con 0x7f139c0b0500 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 >> v1:172.21.15.174:6789/0 conn(0x7f139c0b08e0 legacy=0x7f139c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.908+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 >> v1:172.21.15.64:6789/0 conn(0x7f139c0aeca0 legacy=0x7f139c14b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:34.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: _finish_hunting 0 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 1 monclient: found mon.noname-b 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0accf0 con 0x7f139c0b0500 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: _finish_auth 0 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: _check_auth_tickets 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1398001520 con 0x7f139c0b0500 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a1315640 10 monclient: dump: 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:34.895 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a457b640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.909+0000 7f13a457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1398001680 con 0x7f139c0b0500 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a1315640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a1315640 1 -- 172.21.15.64:0/3913067909 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1398001ad0 con 0x7f139c0b0500 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a1315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a1315640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a1315640 10 monclient: dump: 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a457b640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.910+0000 7f13a457b640 4 set_mon_vals no callback set 2024-03-29T10:48:34.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.911+0000 7f13a457b640 10 monclient: shutdown 2024-03-29T10:48:34.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.911+0000 7f13a457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:34.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.911+0000 7f13a457b640 1 -- 172.21.15.64:0/3913067909 >> v1:172.21.15.64:6790/0 conn(0x7f139c0b0500 legacy=0x7f139c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.911+0000 7f13a457b640 1 -- 172.21.15.64:0/3913067909 shutdown_connections 2024-03-29T10:48:34.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.911+0000 7f13a457b640 1 -- 172.21.15.64:0/3913067909 >> 172.21.15.64:0/3913067909 conn(0x7f139c06b660 msgr2=0x7f139c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:34.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.912+0000 7f13a457b640 1 -- 172.21.15.64:0/3913067909 shutdown_connections 2024-03-29T10:48:34.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.912+0000 7f13a457b640 1 -- 172.21.15.64:0/3913067909 wait complete. 2024-03-29T10:48:34.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.912+0000 7f13a457b640 10 monclient: build_initial_monmap 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.912+0000 7f13a457b640 10 monclient: monmap: 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:34.902 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 1 librados: starting msgr at 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 1 librados: starting objecter 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 1 Processor -- start 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 1 -- start start 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 1 librados: setting wanted keys 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 1 librados: calling monclient init 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.913+0000 7f13a457b640 10 monclient: init 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.914+0000 7f13a457b640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.914+0000 7f13a457b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.914+0000 7f13a457b640 10 monclient(hunting): picked mon.noname-a con 0x7f139c0aeca0 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.914+0000 7f13a457b640 10 monclient(hunting): picked mon.noname-c con 0x7f139c0b0500 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.914+0000 7f13a1b16640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f139c0b0500 0x7f139c1d8750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58382/0 (socket says 172.21.15.64:58382) 2024-03-29T10:48:34.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a457b640 10 monclient(hunting): picked mon.noname-b con 0x7f139c1d8e60 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a457b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c1df110 con 0x7f139c0aeca0 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a457b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c1dbc90 con 0x7f139c1d8e60 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a1b16640 1 -- 172.21.15.64:0/2279134722 learned_addr learned my addr 172.21.15.64:0/2279134722 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c1df630 con 0x7f139c0b0500 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a457b640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f13a457b640 10 monclient(hunting): authenticate will time out at 926.214975s 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 client.?.objecter ms_handle_connect 0x7f139c0b0500 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 client.?.objecter ms_handle_connect 0x7f139c1d8e60 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703379588 0 0) 0x7f139c1df630 con 0x7f139c0b0500 2024-03-29T10:48:34.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1380004eb0 con 0x7f139c0b0500 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089178279 0 0) 0x7f139c1dbc90 con 0x7f139c1d8e60 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c1df630 con 0x7f139c1d8e60 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1280183003 0 0) 0x7f1380004eb0 con 0x7f139c0b0500 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f139c1dbc90 con 0x7f139c0b0500 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f138c001970 con 0x7f139c0b0500 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.906 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.915+0000 7f1392ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.916+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2494343413 0 0) 0x7f139c1dbc90 con 0x7f139c0b0500 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.916+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 >> v1:172.21.15.174:6789/0 conn(0x7f139c1d8e60 legacy=0x7f139c1e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.916+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 >> v1:172.21.15.64:6789/0 conn(0x7f139c0aeca0 legacy=0x7f139c1db580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 1 monclient: found mon.c 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:34.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c1df7c0 con 0x7f139c0b0500 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f1392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 10 monclient: _renew_subs 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c13ab20 con 0x7f139c0b0500 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 10 monclient: _renew_subs 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c06fcf0 con 0x7f139c0b0500 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.917+0000 7f13a457b640 1 librados: init done 2024-03-29T10:48:34.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f138c0047e0 con 0x7f139c0b0500 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f1392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f138c004bf0 con 0x7f139c0b0500 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f1392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f1392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f1392ffd640 10 monclient: dump: 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:34.909 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:34.910 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:34.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.918+0000 7f13a0b14640 4 set_mon_vals no callback set 2024-03-29T10:48:34.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.919+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f138c004e10 con 0x7f139c0b0500 2024-03-29T10:48:34.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.919+0000 7f13a457b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:34.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f13a457b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f13a457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1384005080 con 0x7f139c0b0500 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f138c084b30 con 0x7f139c0b0500 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f1392ffd640 10 client.4334.objecter ms_dispatch 0x7f139c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f1392ffd640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f1392ffd640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f1392ffd640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.920+0000 7f13a2317640 10 monclient: get_auth_request con 0x7f138006dad0 auth_method 0 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.921+0000 7f1392ffd640 10 client.4334.objecter ms_handle_connect 0x7f138006dad0 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.927+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f138c050410 con 0x7f139c0b0500 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.927+0000 7f1392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:34.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:34.927+0000 7f1392ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:35.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.245+0000 7f13a457b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:35.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.245+0000 7f13a457b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.245+0000 7f13a457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.245+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1384005330 con 0x7f139c0b0500 2024-03-29T10:48:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.246+0000 7f1392ffd640 1 -- 172.21.15.64:0/2279134722 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f138c052340 con 0x7f139c0b0500 2024-03-29T10:48:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.246+0000 7f1392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.246+0000 7f1392ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:35.229 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:35.230 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 10 librados: watch_flush enter 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 10 librados: watch_flush exit 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 10 client.4334.objecter successfully canceled tick 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 >> v1:172.21.15.174:6816/2473791532 conn(0x7f138006dad0 legacy=0x7f138006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 10 monclient: shutdown 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 >> v1:172.21.15.64:6790/0 conn(0x7f139c0b0500 legacy=0x7f139c1d8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 shutdown_connections 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.249+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 >> 172.21.15.64:0/2279134722 conn(0x7f139c06b660 msgr2=0x7f139c0b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.250+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 shutdown_connections 2024-03-29T10:48:35.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.250+0000 7f13a457b640 1 -- 172.21.15.64:0/2279134722 wait complete. 2024-03-29T10:48:35.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:35.250+0000 7f13a457b640 1 librados: shutdown 2024-03-29T10:48:35.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:42.053 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:42.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.214+0000 7f36b8814640 10 monclient: get_monmap_and_config 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.214+0000 7f36b8814640 10 monclient: build_initial_monmap 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.214+0000 7f36b8814640 10 monclient: monmap: 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:42.216519+0000 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:42.216519+0000 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:42.198 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:42.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.216+0000 7f36b8814640 1 Processor -- start 2024-03-29T10:48:42.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.216+0000 7f36b8814640 1 -- start start 2024-03-29T10:48:42.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.216+0000 7f36b8814640 10 monclient: init 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient(hunting): picked mon.noname-b con 0x7f36ac0389e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient(hunting): picked mon.noname-c con 0x7f36ac038dc0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient(hunting): picked mon.noname-a con 0x7f36ac030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac0c54c0 con 0x7f36ac030660 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac036430 con 0x7f36ac038dc0 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac039840 con 0x7f36ac0389e0 2024-03-29T10:48:42.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:42.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.217+0000 7f36b8814640 10 monclient(hunting): authenticate will time out at 933.517326s 2024-03-29T10:48:42.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.218+0000 7f36b259c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f36ac038dc0 0x7f36ac0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:52198/0 (socket says 172.21.15.64:52198) 2024-03-29T10:48:42.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.218+0000 7f36b259c640 1 -- 172.21.15.64:0/4215795977 learned_addr learned my addr 172.21.15.64:0/4215795977 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:42.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.219+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239798658 0 0) 0x7f36ac036430 con 0x7f36ac038dc0 2024-03-29T10:48:42.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.219+0000 7f36b1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:42.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.219+0000 7f36b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:42.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.219+0000 7f36b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:42.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.219+0000 7f36b1d9b640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:48:42.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.219+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c003520 con 0x7f36ac038dc0 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2828911796 0 0) 0x7f369c003520 con 0x7f36ac038dc0 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 >> v1:172.21.15.64:6790/0 conn(0x7f36ac0389e0 legacy=0x7f36ac033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 >> v1:172.21.15.64:6789/0 conn(0x7f36ac030660 legacy=0x7f36ac0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 1 monclient: found mon.noname-c 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac035690 con 0x7f36ac038dc0 2024-03-29T10:48:42.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 10 monclient: _finish_auth 0 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f36a8001690 con 0x7f36ac038dc0 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.220+0000 7f36b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.221+0000 7f36b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.221+0000 7f36b1d9b640 10 monclient: dump: 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:42.209 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.221+0000 7f36b8814640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.221+0000 7f36b8814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a8001860 con 0x7f36ac038dc0 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b1d9b640 1 -- 172.21.15.64:0/4215795977 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f36a8001cb0 con 0x7f36ac038dc0 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b1d9b640 10 monclient: dump: 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:42.210 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b8814640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.222+0000 7f36b8814640 4 set_mon_vals no callback set 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 10 monclient: shutdown 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 1 -- 172.21.15.64:0/4215795977 >> v1:172.21.15.174:6789/0 conn(0x7f36ac038dc0 legacy=0x7f36ac0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 1 -- 172.21.15.64:0/4215795977 shutdown_connections 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 1 -- 172.21.15.64:0/4215795977 >> 172.21.15.64:0/4215795977 conn(0x7f36ac005c00 msgr2=0x7f36ac006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:42.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 1 -- 172.21.15.64:0/4215795977 shutdown_connections 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.223+0000 7f36b8814640 1 -- 172.21.15.64:0/4215795977 wait complete. 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 10 monclient: build_initial_monmap 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 10 monclient: monmap: 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:42.212 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 1 librados: starting msgr at 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 1 librados: starting objecter 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 1 Processor -- start 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 1 -- start start 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 1 librados: setting wanted keys 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 1 librados: calling monclient init 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.224+0000 7f36b8814640 10 monclient: init 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient(hunting): picked mon.noname-a con 0x7f36ac030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient(hunting): picked mon.noname-b con 0x7f36ac0389e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient(hunting): picked mon.noname-c con 0x7f36ac02d3c0 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac15d5a0 con 0x7f36ac030660 2024-03-29T10:48:42.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac02e970 con 0x7f36ac0389e0 2024-03-29T10:48:42.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac15c970 con 0x7f36ac02d3c0 2024-03-29T10:48:42.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:42.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.225+0000 7f36b8814640 10 monclient(hunting): authenticate will time out at 933.525055s 2024-03-29T10:48:42.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.227+0000 7f36b359e640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f36ac02d3c0 0x7f36ac02f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50439/0 (socket says 172.21.15.64:50439) 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.227+0000 7f36b359e640 1 -- 172.21.15.64:0/425097280 learned_addr learned my addr 172.21.15.64:0/425097280 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 10 client.?.objecter ms_handle_connect 0x7f36ac02d3c0 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794067622 0 0) 0x7f36ac15c970 con 0x7f36ac02d3c0 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.228+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0004eb0 con 0x7f36ac02d3c0 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692401094 0 0) 0x7f36a0004eb0 con 0x7f36ac02d3c0 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36ac15c970 con 0x7f36ac02d3c0 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f36b404fdd0 con 0x7f36ac02d3c0 2024-03-29T10:48:42.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): dump: 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:42.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 client.?.objecter ms_handle_connect 0x7f36ac0389e0 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060133606 0 0) 0x7f36ac02e970 con 0x7f36ac0389e0 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.229+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0004eb0 con 0x7f36ac0389e0 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4229268970 0 0) 0x7f36ac15c970 con 0x7f36ac02d3c0 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 >> v1:172.21.15.174:6789/0 conn(0x7f36ac0389e0 legacy=0x7f36ac15ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 >> v1:172.21.15.64:6789/0 conn(0x7f36ac030660 legacy=0x7f36ac0c7b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 1 monclient: found mon.c 2024-03-29T10:48:42.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac02fdc0 con 0x7f36ac02d3c0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.230+0000 7f369b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f369b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 10 monclient: _renew_subs 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ac030010 con 0x7f36ac02d3c0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 10 monclient: _renew_subs 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ac02ef70 con 0x7f36ac02d3c0 2024-03-29T10:48:42.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.231+0000 7f36b8814640 1 librados: init done 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b404f400 con 0x7f36ac02d3c0 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f369b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f36b4075f10 con 0x7f36ac02d3c0 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f369b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f369b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f369b7fe640 10 monclient: dump: 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:42.225 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.232+0000 7f36b8814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.233+0000 7f36b8814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.233+0000 7f36b8814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.233+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ac007b90 con 0x7f36ac02d3c0 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.233+0000 7f36b159a640 4 set_mon_vals no callback set 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.234+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f36b4067bf0 con 0x7f36ac02d3c0 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.235+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f36b40ff520 con 0x7f36ac02d3c0 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.235+0000 7f369b7fe640 10 client.4340.objecter ms_dispatch 0x7f36b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.235+0000 7f369b7fe640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.235+0000 7f369b7fe640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.235+0000 7f369b7fe640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T10:48:42.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.237+0000 7f36b2d9d640 10 monclient: get_auth_request con 0x7f36a006daf0 auth_method 0 2024-03-29T10:48:42.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.238+0000 7f369b7fe640 10 client.4340.objecter ms_handle_connect 0x7f36a006daf0 2024-03-29T10:48:42.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.244+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36b40cae00 con 0x7f36ac02d3c0 2024-03-29T10:48:42.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.244+0000 7f369b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:42.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.244+0000 7f369b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:42.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.577+0000 7f36b8814640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.577+0000 7f36b8814640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.577+0000 7f36b8814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.577+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36ac02e7b0 con 0x7f36ac02d3c0 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.579+0000 7f369b7fe640 1 -- 172.21.15.64:0/425097280 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f36b40ccd30 con 0x7f36ac02d3c0 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.579+0000 7f369b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.579+0000 7f369b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:42.562 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.582+0000 7f36b8814640 10 librados: watch_flush enter 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.582+0000 7f36b8814640 10 librados: watch_flush exit 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.582+0000 7f36b8814640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.582+0000 7f36b8814640 10 client.4340.objecter successfully canceled tick 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.583+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 >> v1:172.21.15.174:6816/2473791532 conn(0x7f36a006daf0 legacy=0x7f36a006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.583+0000 7f36b8814640 10 monclient: shutdown 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.583+0000 7f36b8814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:42.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.583+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 >> v1:172.21.15.64:6790/0 conn(0x7f36ac02d3c0 legacy=0x7f36ac02f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:42.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.583+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 shutdown_connections 2024-03-29T10:48:42.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.583+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 >> 172.21.15.64:0/425097280 conn(0x7f36ac005c00 msgr2=0x7f36ac034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:42.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.584+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 shutdown_connections 2024-03-29T10:48:42.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.584+0000 7f36b8814640 1 -- 172.21.15.64:0/425097280 wait complete. 2024-03-29T10:48:42.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:42.584+0000 7f36b8814640 1 librados: shutdown 2024-03-29T10:48:42.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:49.397 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:49.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.565+0000 7f415dd85640 10 monclient: get_monmap_and_config 2024-03-29T10:48:49.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.565+0000 7f415dd85640 10 monclient: build_initial_monmap 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.565+0000 7f415dd85640 10 monclient: monmap: 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:49.566539+0000 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:49.566539+0000 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:49.549 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:49.550 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.569+0000 7f415dd85640 1 Processor -- start 2024-03-29T10:48:49.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.569+0000 7f415dd85640 1 -- start start 2024-03-29T10:48:49.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.569+0000 7f415dd85640 10 monclient: init 2024-03-29T10:48:49.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient(hunting): picked mon.noname-b con 0x7f41580b0860 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient(hunting): picked mon.noname-c con 0x7f415813ef30 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient(hunting): picked mon.noname-a con 0x7f41580af8e0 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580b0c40 con 0x7f41580af8e0 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580aed10 con 0x7f415813ef30 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580afd50 con 0x7f41580b0860 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:49.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.570+0000 7f415dd85640 10 monclient(hunting): authenticate will time out at 940.869270s 2024-03-29T10:48:49.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.572+0000 7f41577fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f41580b0860 0x7f415813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35431/0 (socket says 172.21.15.64:35431) 2024-03-29T10:48:49.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.573+0000 7f41577fe640 1 -- 172.21.15.64:0/555379278 learned_addr learned my addr 172.21.15.64:0/555379278 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:49.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.576+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167770212 0 0) 0x7f41580afd50 con 0x7f41580b0860 2024-03-29T10:48:49.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.576+0000 7f41567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:49.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.576+0000 7f41567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:49.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.576+0000 7f41567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:49.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.576+0000 7f41567fc640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:48:49.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.576+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140003520 con 0x7f41580b0860 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2927335885 0 0) 0x7f4140003520 con 0x7f41580b0860 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 >> v1:172.21.15.174:6789/0 conn(0x7f415813ef30 legacy=0x7f41580af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 >> v1:172.21.15.64:6789/0 conn(0x7f41580af8e0 legacy=0x7f4158149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580ab1b0 con 0x7f41580b0860 2024-03-29T10:48:49.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4148001690 con 0x7f41580b0860 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: dump: 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.563 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:49.564 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:49.564 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:49.564 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41480017f0 con 0x7f41580b0860 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 1 -- 172.21.15.64:0/555379278 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4148001c40 con 0x7f41580b0860 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.579+0000 7f41567fc640 10 monclient: dump: 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:49.565 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.580+0000 7f415dd85640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.580+0000 7f415dd85640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.580+0000 7f415dd85640 4 set_mon_vals no callback set 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 10 monclient: shutdown 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 1 -- 172.21.15.64:0/555379278 >> v1:172.21.15.64:6790/0 conn(0x7f41580b0860 legacy=0x7f415813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 1 -- 172.21.15.64:0/555379278 shutdown_connections 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 1 -- 172.21.15.64:0/555379278 >> 172.21.15.64:0/555379278 conn(0x7f415806b660 msgr2=0x7f415806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 1 -- 172.21.15.64:0/555379278 shutdown_connections 2024-03-29T10:48:49.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.581+0000 7f415dd85640 1 -- 172.21.15.64:0/555379278 wait complete. 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.582+0000 7f415dd85640 10 monclient: build_initial_monmap 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.582+0000 7f415dd85640 10 monclient: monmap: 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:49.567 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.582+0000 7f415dd85640 1 librados: starting msgr at 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.582+0000 7f415dd85640 1 librados: starting objecter 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.583+0000 7f415dd85640 1 Processor -- start 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.583+0000 7f415dd85640 1 -- start start 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.583+0000 7f415dd85640 1 librados: setting wanted keys 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.583+0000 7f415dd85640 1 librados: calling monclient init 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.583+0000 7f415dd85640 10 monclient: init 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.584+0000 7f415dd85640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.584+0000 7f415dd85640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.584+0000 7f415dd85640 10 monclient(hunting): picked mon.noname-b con 0x7f41580af8e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 10 monclient(hunting): picked mon.noname-c con 0x7f41580b0860 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 10 monclient(hunting): picked mon.noname-a con 0x7f415813ef30 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:49.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4158133150 con 0x7f415813ef30 2024-03-29T10:48:49.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4158139fb0 con 0x7f41580af8e0 2024-03-29T10:48:49.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41581373b0 con 0x7f41580b0860 2024-03-29T10:48:49.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:49.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.585+0000 7f415dd85640 10 monclient(hunting): authenticate will time out at 940.884085s 2024-03-29T10:48:49.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.588+0000 7f4156ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f41580b0860 0x7f4158132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35541/0 (socket says 172.21.15.64:35541) 2024-03-29T10:48:49.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.588+0000 7f4156ffd640 1 -- 172.21.15.64:0/1535795309 learned_addr learned my addr 172.21.15.64:0/1535795309 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:49.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.589+0000 7f413ffff640 10 client.?.objecter ms_handle_connect 0x7f415813ef30 2024-03-29T10:48:49.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.589+0000 7f413ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:49.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.589+0000 7f413ffff640 10 client.?.objecter ms_handle_connect 0x7f41580b0860 2024-03-29T10:48:49.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.589+0000 7f413ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:49.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941422363 0 0) 0x7f4158133150 con 0x7f415813ef30 2024-03-29T10:48:49.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:49.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:49.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:49.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:48:49.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144004eb0 con 0x7f415813ef30 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707444680 0 0) 0x7f41581373b0 con 0x7f41580b0860 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4158133150 con 0x7f41580b0860 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 80576113 0 0) 0x7f4158133150 con 0x7f41580b0860 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41581373b0 con 0x7f41580b0860 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f414c001800 con 0x7f41580b0860 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:49.587 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.593+0000 7f413ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.594+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 822593136 0 0) 0x7f4144004eb0 con 0x7f415813ef30 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.594+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4158133150 con 0x7f415813ef30 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.594+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4150001bf0 con 0x7f415813ef30 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.594+0000 7f413ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:49.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.594+0000 7f413ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.594+0000 7f413ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 10 client.?.objecter ms_handle_connect 0x7f41580af8e0 2024-03-29T10:48:49.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766847442 0 0) 0x7f4158139fb0 con 0x7f41580af8e0 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.595+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144004eb0 con 0x7f41580af8e0 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4024700418 0 0) 0x7f4144004eb0 con 0x7f41580af8e0 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4158139fb0 con 0x7f41580af8e0 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4148001b00 con 0x7f41580af8e0 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:49.590 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.596+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2718505162 0 0) 0x7f4158139fb0 con 0x7f41580af8e0 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 >> v1:172.21.15.64:6790/0 conn(0x7f41580b0860 legacy=0x7f4158132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 >> v1:172.21.15.64:6789/0 conn(0x7f415813ef30 legacy=0x7f4158132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:49.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 1 monclient: found mon.b 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158138310 con 0x7f41580af8e0 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f413ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 5 monclient: authenticate success, global_id 4339 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 10 monclient: _renew_subs 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415813bc40 con 0x7f41580af8e0 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 10 monclient: _renew_subs 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:48:49.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4158139010 con 0x7f41580af8e0 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.597+0000 7f415dd85640 1 librados: init done 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.598+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4148001c60 con 0x7f41580af8e0 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.598+0000 7f413ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.598+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4148004d60 con 0x7f41580af8e0 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.598+0000 7f413ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.598+0000 7f413ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.598+0000 7f413ffff640 10 monclient: dump: 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:49.593 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.599+0000 7f415dd85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.599+0000 7f4155ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.600+0000 7f415dd85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.600+0000 7f415dd85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.600+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158075f10 con 0x7f41580af8e0 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.601+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f414801ad30 con 0x7f41580af8e0 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f4156ffd640 10 monclient: get_auth_request con 0x7f414407f220 auth_method 0 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f4148084a40 con 0x7f41580af8e0 2024-03-29T10:48:49.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f413ffff640 10 client.4339.objecter ms_dispatch 0x7f4158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:49.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f413ffff640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:49.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f413ffff640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:49.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f413ffff640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T10:48:49.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.602+0000 7f413ffff640 10 client.4339.objecter ms_handle_connect 0x7f414407f220 2024-03-29T10:48:49.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.611+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4148050320 con 0x7f41580af8e0 2024-03-29T10:48:49.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.611+0000 7f413ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:49.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.611+0000 7f413ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:49.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.943+0000 7f415dd85640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:49.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.943+0000 7f415dd85640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:49.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.943+0000 7f415dd85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:48:49.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.943+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4158072330 con 0x7f41580af8e0 2024-03-29T10:48:49.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.945+0000 7f413ffff640 1 -- 172.21.15.64:0/1535795309 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f4148052250 con 0x7f41580af8e0 2024-03-29T10:48:49.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.945+0000 7f413ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:49.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.945+0000 7f413ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:49.930 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:49.930 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:49.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 10 librados: watch_flush enter 2024-03-29T10:48:49.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 10 librados: watch_flush exit 2024-03-29T10:48:49.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T10:48:49.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 10 client.4339.objecter successfully canceled tick 2024-03-29T10:48:49.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 >> v1:172.21.15.174:6816/2473791532 conn(0x7f414407f220 legacy=0x7f41440816e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 10 monclient: shutdown 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 >> v1:172.21.15.174:6789/0 conn(0x7f41580af8e0 legacy=0x7f41581ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 shutdown_connections 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 >> 172.21.15.64:0/1535795309 conn(0x7f415806b660 msgr2=0x7f415813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 shutdown_connections 2024-03-29T10:48:49.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.950+0000 7f415dd85640 1 -- 172.21.15.64:0/1535795309 wait complete. 2024-03-29T10:48:49.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:49.954+0000 7f415dd85640 1 librados: shutdown 2024-03-29T10:48:49.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:56.763 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:56.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.929+0000 7f32d8e8f640 10 monclient: get_monmap_and_config 2024-03-29T10:48:56.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.929+0000 7f32d8e8f640 10 monclient: build_initial_monmap 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.929+0000 7f32d8e8f640 10 monclient: monmap: 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:48:56.930552+0000 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:48:56.930552+0000 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:48:56.912 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.932+0000 7f32d8e8f640 1 Processor -- start 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.932+0000 7f32d8e8f640 1 -- start start 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.932+0000 7f32d8e8f640 10 monclient: init 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f32d40b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f32d4071eb0 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f32d4143d10 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:56.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d406ef80 con 0x7f32d4071eb0 2024-03-29T10:48:56.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d406f260 con 0x7f32d4143d10 2024-03-29T10:48:56.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40b40c0 con 0x7f32d40b1f60 2024-03-29T10:48:56.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:56.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.933+0000 7f32d8e8f640 10 monclient(hunting): authenticate will time out at 948.232181s 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d1d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f32d4071eb0 0x7f32d4146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43087/0 (socket says 172.21.15.64:43087) 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d1d9b640 1 -- 172.21.15.64:0/551976342 learned_addr learned my addr 172.21.15.64:0/551976342 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577917295 0 0) 0x7f32d406ef80 con 0x7f32d4071eb0 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d159a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d159a640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.936+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32bc003520 con 0x7f32d4071eb0 2024-03-29T10:48:56.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.937+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 379077247 0 0) 0x7f32bc003520 con 0x7f32d4071eb0 2024-03-29T10:48:56.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.937+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 >> v1:172.21.15.64:6790/0 conn(0x7f32d40b1f60 legacy=0x7f32d4143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:56.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 >> v1:172.21.15.174:6789/0 conn(0x7f32d4143d10 legacy=0x7f32d4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:56.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 1 monclient: found mon.noname-a 2024-03-29T10:48:56.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4155d70 con 0x7f32d4071eb0 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: _finish_auth 0 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32cc001a10 con 0x7f32d4071eb0 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.940+0000 7f32d159a640 10 monclient: dump: 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.924 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.941+0000 7f32d8e8f640 5 monclient: authenticate success, global_id 4317 2024-03-29T10:48:56.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.941+0000 7f32d8e8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32cc001c70 con 0x7f32d4071eb0 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d159a640 1 -- 172.21.15.64:0/551976342 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32cc002080 con 0x7f32d4071eb0 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d159a640 10 monclient: dump: 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.927 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.928 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.928 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.928 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:56.928 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:56.928 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:56.928 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d8e8f640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:56.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.944+0000 7f32d8e8f640 4 set_mon_vals no callback set 2024-03-29T10:48:56.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.945+0000 7f32d8e8f640 10 monclient: shutdown 2024-03-29T10:48:56.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.945+0000 7f32d8e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:56.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.945+0000 7f32d8e8f640 1 -- 172.21.15.64:0/551976342 >> v1:172.21.15.64:6789/0 conn(0x7f32d4071eb0 legacy=0x7f32d4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.945+0000 7f32d8e8f640 1 -- 172.21.15.64:0/551976342 shutdown_connections 2024-03-29T10:48:56.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.945+0000 7f32d8e8f640 1 -- 172.21.15.64:0/551976342 >> 172.21.15.64:0/551976342 conn(0x7f32d4066a00 msgr2=0x7f32d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:56.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.949+0000 7f32d8e8f640 1 -- 172.21.15.64:0/551976342 shutdown_connections 2024-03-29T10:48:56.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.949+0000 7f32d8e8f640 1 -- 172.21.15.64:0/551976342 wait complete. 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.950+0000 7f32d8e8f640 10 monclient: build_initial_monmap 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.950+0000 7f32d8e8f640 10 monclient: monmap: 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.932 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:48:56.933 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:48:56.933 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:48:56.933 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.950+0000 7f32d8e8f640 1 librados: starting msgr at 2024-03-29T10:48:56.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.950+0000 7f32d8e8f640 1 librados: starting objecter 2024-03-29T10:48:56.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 1 Processor -- start 2024-03-29T10:48:56.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 1 -- start start 2024-03-29T10:48:56.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 1 librados: setting wanted keys 2024-03-29T10:48:56.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 1 librados: calling monclient init 2024-03-29T10:48:56.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 10 monclient: init 2024-03-29T10:48:56.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:56.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.951+0000 7f32d8e8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:56.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.952+0000 7f32d8e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f32d40b1f60 addr v1:172.21.15.174:6789/0 2024-03-29T10:48:56.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.955+0000 7f32d259c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f32d40b1f60 0x7f32d4147ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:50689/0 (socket says 172.21.15.64:50689) 2024-03-29T10:48:56.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.957+0000 7f32d8e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f32d4143d10 addr v1:172.21.15.64:6790/0 2024-03-29T10:48:56.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.957+0000 7f32d259c640 1 -- 172.21.15.64:0/2067853332 learned_addr learned my addr 172.21.15.64:0/2067853332 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32c2ffd640 10 client.?.objecter ms_handle_connect 0x7f32d40b1f60 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32d8e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f32d4154d40 addr v1:172.21.15.64:6789/0 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d414b2d0 con 0x7f32d4154d40 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d414b5b0 con 0x7f32d40b1f60 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d4148490 con 0x7f32d4143d10 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32d8e8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:56.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.960+0000 7f32d8e8f640 10 monclient(hunting): authenticate will time out at 948.259611s 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 client.?.objecter ms_handle_connect 0x7f32d4154d40 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 client.?.objecter ms_handle_connect 0x7f32d4143d10 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706742734 0 0) 0x7f32d414b5b0 con 0x7f32d40b1f60 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b4004eb0 con 0x7f32d40b1f60 2024-03-29T10:48:56.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45410142 0 0) 0x7f32d4148490 con 0x7f32d4143d10 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d414b5b0 con 0x7f32d4143d10 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709649979 0 0) 0x7f32d414b2d0 con 0x7f32d4154d40 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:48:56.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.965+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d4148490 con 0x7f32d4154d40 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3196411061 0 0) 0x7f32d414b5b0 con 0x7f32d4143d10 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d414b2d0 con 0x7f32d4143d10 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32cc001830 con 0x7f32d4143d10 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.949 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3077017694 0 0) 0x7f32b4004eb0 con 0x7f32d40b1f60 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d414b5b0 con 0x7f32d40b1f60 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32c4001940 con 0x7f32d40b1f60 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:56.950 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1709175849 0 0) 0x7f32d414b2d0 con 0x7f32d4143d10 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 >> v1:172.21.15.174:6789/0 conn(0x7f32d40b1f60 legacy=0x7f32d4147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 >> v1:172.21.15.64:6789/0 conn(0x7f32d4154d40 legacy=0x7f32d4149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 monclient: found mon.c 2024-03-29T10:48:56.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:56.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:56.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4152750 con 0x7f32d4143d10 2024-03-29T10:48:56.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:56.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:56.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.966+0000 7f32c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 10 monclient: _renew_subs 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d4152a20 con 0x7f32d4143d10 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 10 monclient: _renew_subs 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d4149c20 con 0x7f32d4143d10 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.967+0000 7f32d8e8f640 1 librados: init done 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.969+0000 7f32d8e8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.969+0000 7f32d8e8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.969+0000 7f32d8e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:56.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.969+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d4138a00 con 0x7f32d4143d10 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32cc0015f0 con 0x7f32d4143d10 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f32cc004fb0 con 0x7f32d4143d10 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 10 monclient: dump: 2024-03-29T10:48:56.956 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:48:56.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.974+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f32cc01af80 con 0x7f32d4143d10 2024-03-29T10:48:56.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.975+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f32cc084e00 con 0x7f32d4143d10 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.975+0000 7f32c2ffd640 10 client.4352.objecter ms_dispatch 0x7f32d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.975+0000 7f32c2ffd640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.976+0000 7f32d0d99640 4 set_mon_vals no callback set 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.976+0000 7f32c2ffd640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.976+0000 7f32d259c640 10 monclient: get_auth_request con 0x7f32b406e270 auth_method 0 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.976+0000 7f32c2ffd640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T10:48:56.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.976+0000 7f32c2ffd640 10 client.4352.objecter ms_handle_connect 0x7f32b406e270 2024-03-29T10:48:56.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.981+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32cc050760 con 0x7f32d4143d10 2024-03-29T10:48:56.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.981+0000 7f32c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:56.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:56.981+0000 7f32c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:57.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.299+0000 7f32d8e8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:57.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.299+0000 7f32d8e8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:57.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.299+0000 7f32d8e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:48:57.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.299+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32d414bad0 con 0x7f32d4143d10 2024-03-29T10:48:57.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.300+0000 7f32c2ffd640 1 -- 172.21.15.64:0/2067853332 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f32d414bad0 con 0x7f32d4143d10 2024-03-29T10:48:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.301+0000 7f32c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:57.283 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:48:57.283 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:57.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.301+0000 7f32c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.304+0000 7f32d8e8f640 10 librados: watch_flush enter 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.304+0000 7f32d8e8f640 10 librados: watch_flush exit 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.304+0000 7f32d8e8f640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.304+0000 7f32d8e8f640 10 client.4352.objecter successfully canceled tick 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.304+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 >> v1:172.21.15.174:6816/2473791532 conn(0x7f32b406e270 legacy=0x7f32b4070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.305+0000 7f32d8e8f640 10 monclient: shutdown 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.305+0000 7f32d8e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.305+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 >> v1:172.21.15.64:6790/0 conn(0x7f32d4143d10 legacy=0x7f32d4152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.305+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 shutdown_connections 2024-03-29T10:48:57.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.305+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 >> 172.21.15.64:0/2067853332 conn(0x7f32d4066a00 msgr2=0x7f32d40b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:57.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.306+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 shutdown_connections 2024-03-29T10:48:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.307+0000 7f32d8e8f640 1 -- 172.21.15.64:0/2067853332 wait complete. 2024-03-29T10:48:57.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:48:57.308+0000 7f32d8e8f640 1 librados: shutdown 2024-03-29T10:48:57.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:04.114 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:04.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.282+0000 7fea0bf3a640 10 monclient: get_monmap_and_config 2024-03-29T10:49:04.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.282+0000 7fea0bf3a640 10 monclient: build_initial_monmap 2024-03-29T10:49:04.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.282+0000 7fea0bf3a640 10 monclient: monmap: 2024-03-29T10:49:04.264 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:04.264 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:04.282975+0000 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:04.282975+0000 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.282+0000 7fea0bf3a640 1 Processor -- start 2024-03-29T10:49:04.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.283+0000 7fea0bf3a640 1 -- start start 2024-03-29T10:49:04.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.283+0000 7fea0bf3a640 10 monclient: init 2024-03-29T10:49:04.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:04.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:04.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient(hunting): picked mon.noname-c con 0x7fea040b0770 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient(hunting): picked mon.noname-a con 0x7fea0414a700 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient(hunting): picked mon.noname-b con 0x7fea040ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea040aba70 con 0x7fea0414a700 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea040aecb0 con 0x7fea040b0770 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea040af150 con 0x7fea040ac5a0 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:04.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.284+0000 7fea0bf3a640 10 monclient(hunting): authenticate will time out at 955.582991s 2024-03-29T10:49:04.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea094d5640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fea0414a700 0x7fea040abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33287/0 (socket says 172.21.15.64:33287) 2024-03-29T10:49:04.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea094d5640 1 -- 172.21.15.64:0/1989732907 learned_addr learned my addr 172.21.15.64:0/1989732907 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:04.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578927303 0 0) 0x7fea040aba70 con 0x7fea0414a700 2024-03-29T10:49:04.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea08cd4640 10 monclient(hunting): _negotiate 2024-03-29T10:49:04.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea08cd4640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:04.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea08cd4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:04.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea08cd4640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:49:04.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.288+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f0003520 con 0x7fea0414a700 2024-03-29T10:49:04.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.289+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2716374301 0 0) 0x7fe9f0003520 con 0x7fea0414a700 2024-03-29T10:49:04.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.289+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 >> v1:172.21.15.64:6790/0 conn(0x7fea040ac5a0 legacy=0x7fea04151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 >> v1:172.21.15.174:6789/0 conn(0x7fea040b0770 legacy=0x7fea04149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: _finish_hunting 0 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 1 monclient: found mon.noname-a 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea040ac9d0 con 0x7fea0414a700 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: _finish_auth 0 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: _check_auth_tickets 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:04.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea00001670 con 0x7fea0414a700 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea08cd4640 10 monclient: dump: 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:04.277 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea0bf3a640 5 monclient: authenticate success, global_id 4323 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.290+0000 7fea0bf3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.291+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea000018d0 con 0x7fea0414a700 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.291+0000 7fea08cd4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.291+0000 7fea0bf3a640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.291+0000 7fea0bf3a640 4 set_mon_vals no callback set 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea08cd4640 1 -- 172.21.15.64:0/1989732907 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea00001d20 con 0x7fea0414a700 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea08cd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea08cd4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea08cd4640 10 monclient: dump: 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.278 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 10 monclient: shutdown 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 1 -- 172.21.15.64:0/1989732907 >> v1:172.21.15.64:6789/0 conn(0x7fea0414a700 legacy=0x7fea040abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 1 -- 172.21.15.64:0/1989732907 shutdown_connections 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 1 -- 172.21.15.64:0/1989732907 >> 172.21.15.64:0/1989732907 conn(0x7fea0406b660 msgr2=0x7fea0406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 1 -- 172.21.15.64:0/1989732907 shutdown_connections 2024-03-29T10:49:04.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.292+0000 7fea0bf3a640 1 -- 172.21.15.64:0/1989732907 wait complete. 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 10 monclient: build_initial_monmap 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 10 monclient: monmap: 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 1 librados: starting msgr at 2024-03-29T10:49:04.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 1 librados: starting objecter 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 1 Processor -- start 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 1 -- start start 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 1 librados: setting wanted keys 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 1 librados: calling monclient init 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.293+0000 7fea0bf3a640 10 monclient: init 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient(hunting): picked mon.noname-a con 0x7fea040ac5a0 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:04.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient(hunting): picked mon.noname-c con 0x7fea040b0770 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient(hunting): picked mon.noname-b con 0x7fea041dc040 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea041db4a0 con 0x7fea040ac5a0 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea04127430 con 0x7fea041dc040 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea041239d0 con 0x7fea040b0770 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.294+0000 7fea0bf3a640 10 monclient(hunting): authenticate will time out at 955.593560s 2024-03-29T10:49:04.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.300+0000 7fea094d5640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fea040b0770 0x7fea041db930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53543/0 (socket says 172.21.15.64:53543) 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.300+0000 7fea094d5640 1 -- 172.21.15.64:0/3644968179 learned_addr learned my addr 172.21.15.64:0/3644968179 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 10 client.?.objecter ms_handle_connect 0x7fea040b0770 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873950719 0 0) 0x7fea041239d0 con 0x7fea040b0770 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:04.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:49:04.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.301+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e8004eb0 con 0x7fea040b0770 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 client.?.objecter ms_handle_connect 0x7fea040ac5a0 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289655791 0 0) 0x7fea041db4a0 con 0x7fea040ac5a0 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:49:04.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea041239d0 con 0x7fea040ac5a0 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 client.?.objecter ms_handle_connect 0x7fea041dc040 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328194439 0 0) 0x7fea041239d0 con 0x7fea040ac5a0 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea041db4a0 con 0x7fea040ac5a0 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.303+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3516720623 0 0) 0x7fe9e8004eb0 con 0x7fea040b0770 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea041239d0 con 0x7fea040b0770 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9f40017d0 con 0x7fea040ac5a0 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): dump: 2024-03-29T10:49:04.286 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea00001630 con 0x7fea040b0770 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:04.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): dump: 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816329053 0 0) 0x7fea04127430 con 0x7fea041dc040 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:04.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:04.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:49:04.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e8004eb0 con 0x7fea041dc040 2024-03-29T10:49:04.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3586609859 0 0) 0x7fea041db4a0 con 0x7fea040ac5a0 2024-03-29T10:49:04.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 >> v1:172.21.15.64:6790/0 conn(0x7fea040b0770 legacy=0x7fea041db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:04.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 >> v1:172.21.15.174:6789/0 conn(0x7fea041dc040 legacy=0x7fea041e6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 monclient: found mon.a 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04128610 con 0x7fea040ac5a0 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fe9fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fea0bf3a640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fea0bf3a640 10 monclient: _renew_subs 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fea0bf3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:04.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.304+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea041261b0 con 0x7fea040ac5a0 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.305+0000 7fea0bf3a640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.305+0000 7fea0bf3a640 10 monclient: _renew_subs 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.305+0000 7fea0bf3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.305+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea041e7750 con 0x7fea040ac5a0 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.305+0000 7fea0bf3a640 1 librados: init done 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.306+0000 7fea0bf3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.306+0000 7fea0bf3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.306+0000 7fea0bf3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:04.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.306+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea040ab560 con 0x7fea040ac5a0 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.308+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9f4001930 con 0x7fea040ac5a0 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.308+0000 7fe9fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.308+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9f4003790 con 0x7fea040ac5a0 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.308+0000 7fe9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.308+0000 7fe9fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.308+0000 7fe9fa7fc640 10 monclient: dump: 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:04.292 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:04.293 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:04.293 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:04.293 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:04.293 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.311+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe9f40038f0 con 0x7fea040ac5a0 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.311+0000 7fe9fbfff640 4 set_mon_vals no callback set 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fea094d5640 10 monclient: get_auth_request con 0x7fe9e806dfe0 auth_method 0 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe9f40836a0 con 0x7fea040ac5a0 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fe9fa7fc640 10 client.4326.objecter ms_dispatch 0x7fea04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fe9fa7fc640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fe9fa7fc640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fe9fa7fc640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T10:49:04.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.312+0000 7fe9fa7fc640 10 client.4326.objecter ms_handle_connect 0x7fe9e806dfe0 2024-03-29T10:49:04.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.317+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9f404efe0 con 0x7fea040ac5a0 2024-03-29T10:49:04.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.318+0000 7fe9fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:04.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.318+0000 7fe9fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:04.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.648+0000 7fea0bf3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:04.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.648+0000 7fea0bf3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:04.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.648+0000 7fea0bf3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:04.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.648+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea04075f10 con 0x7fea040ac5a0 2024-03-29T10:49:04.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.653+0000 7fe9fa7fc640 1 -- 172.21.15.64:0/3644968179 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe9f4050f10 con 0x7fea040ac5a0 2024-03-29T10:49:04.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.653+0000 7fe9fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:04.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.653+0000 7fe9fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:04.636 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:04.636 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.656+0000 7fea0bf3a640 10 librados: watch_flush enter 2024-03-29T10:49:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.656+0000 7fea0bf3a640 10 librados: watch_flush exit 2024-03-29T10:49:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.656+0000 7fea0bf3a640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T10:49:04.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.656+0000 7fea0bf3a640 10 client.4326.objecter successfully canceled tick 2024-03-29T10:49:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe9e806dfe0 legacy=0x7fe9e80704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 10 monclient: shutdown 2024-03-29T10:49:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 >> v1:172.21.15.64:6789/0 conn(0x7fea040ac5a0 legacy=0x7fea04126d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 shutdown_connections 2024-03-29T10:49:04.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 >> 172.21.15.64:0/3644968179 conn(0x7fea0406b660 msgr2=0x7fea041494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 shutdown_connections 2024-03-29T10:49:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 -- 172.21.15.64:0/3644968179 wait complete. 2024-03-29T10:49:04.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:04.657+0000 7fea0bf3a640 1 librados: shutdown 2024-03-29T10:49:04.655 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:11.462 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:11.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.643+0000 7fe54cd72640 10 monclient: get_monmap_and_config 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.643+0000 7fe54cd72640 10 monclient: build_initial_monmap 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.643+0000 7fe54cd72640 10 monclient: monmap: 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:11.644274+0000 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:11.644274+0000 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:11.626 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:11.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.644+0000 7fe54cd72640 1 Processor -- start 2024-03-29T10:49:11.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.644+0000 7fe54cd72640 1 -- start start 2024-03-29T10:49:11.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.644+0000 7fe54cd72640 10 monclient: init 2024-03-29T10:49:11.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.645+0000 7fe54cd72640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:11.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.645+0000 7fe54cd72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:11.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.645+0000 7fe54cd72640 10 monclient(hunting): picked mon.noname-c con 0x7fe5480b0500 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:11.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 10 monclient(hunting): picked mon.noname-b con 0x7fe5480b08e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:11.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54659c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe5480b0500 0x7fe548072050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:50385/0 (socket says 172.21.15.64:50385) 2024-03-29T10:49:11.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 10 monclient(hunting): picked mon.noname-a con 0x7fe5480aec70 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:11.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480ae7e0 con 0x7fe5480aec70 2024-03-29T10:49:11.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54813daf0 con 0x7fe5480b0500 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480accf0 con 0x7fe5480b08e0 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54cd72640 10 monclient(hunting): authenticate will time out at 962.948431s 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.649+0000 7fe54659c640 1 -- 172.21.15.64:0/2277943858 learned_addr learned my addr 172.21.15.64:0/2277943858 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743486285 0 0) 0x7fe5480accf0 con 0x7fe5480b08e0 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:11.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534003520 con 0x7fe5480b08e0 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278789688 0 0) 0x7fe54813daf0 con 0x7fe5480b0500 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5480accf0 con 0x7fe5480b0500 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048447338 0 0) 0x7fe5480accf0 con 0x7fe5480b0500 2024-03-29T10:49:11.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 >> v1:172.21.15.64:6790/0 conn(0x7fe5480b08e0 legacy=0x7fe54813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:11.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.650+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 >> v1:172.21.15.64:6789/0 conn(0x7fe5480aec70 legacy=0x7fe54813ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 1 monclient: found mon.noname-c 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548140620 con 0x7fe5480b0500 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: _finish_auth 0 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe530001670 con 0x7fe5480b0500 2024-03-29T10:49:11.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54559a640 10 monclient: dump: 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54cd72640 5 monclient: authenticate success, global_id 4351 2024-03-29T10:49:11.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.652+0000 7fe54cd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.653+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5300017d0 con 0x7fe5480b0500 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.653+0000 7fe54559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.653+0000 7fe54cd72640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.653+0000 7fe54cd72640 4 set_mon_vals no callback set 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.654+0000 7fe54cd72640 10 monclient: shutdown 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.654+0000 7fe54cd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:11.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.654+0000 7fe54cd72640 1 -- 172.21.15.64:0/2277943858 >> v1:172.21.15.174:6789/0 conn(0x7fe5480b0500 legacy=0x7fe548072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:11.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.654+0000 7fe54559a640 1 -- 172.21.15.64:0/2277943858 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe530001be0 con 0x7fe5480b0500 2024-03-29T10:49:11.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.654+0000 7fe54559a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:49:11.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.657+0000 7fe54cd72640 1 -- 172.21.15.64:0/2277943858 shutdown_connections 2024-03-29T10:49:11.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.657+0000 7fe54cd72640 1 -- 172.21.15.64:0/2277943858 >> 172.21.15.64:0/2277943858 conn(0x7fe54806b660 msgr2=0x7fe54806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:11.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.657+0000 7fe54cd72640 1 -- 172.21.15.64:0/2277943858 shutdown_connections 2024-03-29T10:49:11.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.657+0000 7fe54cd72640 1 -- 172.21.15.64:0/2277943858 wait complete. 2024-03-29T10:49:11.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.659+0000 7fe54cd72640 10 monclient: build_initial_monmap 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.659+0000 7fe54cd72640 10 monclient: monmap: 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:11.641 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:11.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.659+0000 7fe54cd72640 1 librados: starting msgr at 2024-03-29T10:49:11.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.660+0000 7fe54cd72640 1 librados: starting objecter 2024-03-29T10:49:11.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.660+0000 7fe54cd72640 1 Processor -- start 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.660+0000 7fe54cd72640 1 -- start start 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.660+0000 7fe54cd72640 1 librados: setting wanted keys 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.660+0000 7fe54cd72640 1 librados: calling monclient init 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.660+0000 7fe54cd72640 10 monclient: init 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.661+0000 7fe54cd72640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.661+0000 7fe54cd72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:11.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.661+0000 7fe54cd72640 10 monclient(hunting): picked mon.noname-a con 0x7fe5480aec70 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 10 monclient(hunting): picked mon.noname-c con 0x7fe5480b0500 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 10 monclient(hunting): picked mon.noname-b con 0x7fe5480b35a0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480b3ec0 con 0x7fe5480aec70 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5481374a0 con 0x7fe5480b35a0 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe548138310 con 0x7fe5480b0500 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:11.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.662+0000 7fe54cd72640 10 monclient(hunting): authenticate will time out at 962.961506s 2024-03-29T10:49:11.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe545d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe5480b0500 0x7fe548139d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54417/0 (socket says 172.21.15.64:54417) 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe545d9b640 1 -- 172.21.15.64:0/1024456566 learned_addr learned my addr 172.21.15.64:0/1024456566 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 10 client.?.objecter ms_handle_connect 0x7fe5480b0500 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320487116 0 0) 0x7fe548138310 con 0x7fe5480b0500 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.668+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe528004eb0 con 0x7fe5480b0500 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009755778 0 0) 0x7fe528004eb0 con 0x7fe5480b0500 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe548138310 con 0x7fe5480b0500 2024-03-29T10:49:11.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe538001cf0 con 0x7fe5480b0500 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 10 monclient(hunting): dump: 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:11.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.669+0000 7fe53effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:11.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.670+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4094674369 0 0) 0x7fe548138310 con 0x7fe5480b0500 2024-03-29T10:49:11.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.670+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 >> v1:172.21.15.174:6789/0 conn(0x7fe5480b35a0 legacy=0x7fe5481de650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:11.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.670+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 >> v1:172.21.15.64:6789/0 conn(0x7fe5480aec70 legacy=0x7fe5481d62b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 1 monclient: found mon.c 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480b4350 con 0x7fe5480b0500 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 10 monclient: _finish_auth 0 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe53effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 10 monclient: _renew_subs 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5480b3a50 con 0x7fe5480b0500 2024-03-29T10:49:11.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 10 monclient: _renew_subs 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5481d6ba0 con 0x7fe5480b0500 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.671+0000 7fe54cd72640 1 librados: init done 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe538001ff0 con 0x7fe5480b0500 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe53effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe538003de0 con 0x7fe5480b0500 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe53effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe53effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe53effd640 10 monclient: dump: 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:11.656 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:11.657 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:11.657 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:11.658 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:11.658 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:11.658 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:11.658 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:11.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.672+0000 7fe544d99640 4 set_mon_vals no callback set 2024-03-29T10:49:11.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.674+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe538003f40 con 0x7fe5480b0500 2024-03-29T10:49:11.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.674+0000 7fe54cd72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:11.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.674+0000 7fe54cd72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:11.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.674+0000 7fe54cd72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.674+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c005080 con 0x7fe5480b0500 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe538084ea0 con 0x7fe5480b0500 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe53effd640 10 client.4361.objecter ms_dispatch 0x7fe548000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe53effd640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe53effd640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe53effd640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe54659c640 10 monclient: get_auth_request con 0x7fe52806dc70 auth_method 0 2024-03-29T10:49:11.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.676+0000 7fe53effd640 10 client.4361.objecter ms_handle_connect 0x7fe52806dc70 2024-03-29T10:49:11.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.684+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe538050780 con 0x7fe5480b0500 2024-03-29T10:49:11.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.685+0000 7fe53effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:11.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:11.685+0000 7fe53effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:11.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.010+0000 7fe54cd72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:11.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.010+0000 7fe54cd72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:11.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.010+0000 7fe54cd72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:11.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.010+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe52c005330 con 0x7fe5480b0500 2024-03-29T10:49:11.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.011+0000 7fe53effd640 1 -- 172.21.15.64:0/1024456566 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe5380526b0 con 0x7fe5480b0500 2024-03-29T10:49:11.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.011+0000 7fe53effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:11.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.011+0000 7fe53effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:11.997 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:11.997 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.015+0000 7fe54cd72640 10 librados: watch_flush enter 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 10 librados: watch_flush exit 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 10 client.4361.objecter successfully canceled tick 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe52806dc70 legacy=0x7fe528070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 10 monclient: shutdown 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 >> v1:172.21.15.64:6790/0 conn(0x7fe5480b0500 legacy=0x7fe548139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 shutdown_connections 2024-03-29T10:49:11.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 >> 172.21.15.64:0/1024456566 conn(0x7fe54806b660 msgr2=0x7fe54813de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.016+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 shutdown_connections 2024-03-29T10:49:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.017+0000 7fe54cd72640 1 -- 172.21.15.64:0/1024456566 wait complete. 2024-03-29T10:49:11.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:12.017+0000 7fe54cd72640 1 librados: shutdown 2024-03-29T10:49:12.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:18.826 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:18.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.986+0000 7f08b6431640 10 monclient: get_monmap_and_config 2024-03-29T10:49:18.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.986+0000 7f08b6431640 10 monclient: build_initial_monmap 2024-03-29T10:49:18.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.986+0000 7f08b6431640 10 monclient: monmap: 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:18.987605+0000 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:18.987605+0000 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:18.969 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:18.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.987+0000 7f08b6431640 1 Processor -- start 2024-03-29T10:49:18.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.987+0000 7f08b6431640 1 -- start start 2024-03-29T10:49:18.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.987+0000 7f08b6431640 10 monclient: init 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient(hunting): picked mon.noname-a con 0x7f08b00b1f60 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient(hunting): picked mon.noname-c con 0x7f08b0071eb0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient(hunting): picked mon.noname-b con 0x7f08b0143d10 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b006ef80 con 0x7f08b00b1f60 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b006f260 con 0x7f08b0071eb0 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b40c0 con 0x7f08b0143d10 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:18.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.988+0000 7f08b6431640 10 monclient(hunting): authenticate will time out at 970.287847s 2024-03-29T10:49:18.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.989+0000 7f08b49ce640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f08b0143d10 0x7f08b0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33346/0 (socket says 172.21.15.64:33346) 2024-03-29T10:49:18.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.989+0000 7f08b49ce640 1 -- 172.21.15.64:0/2385853627 learned_addr learned my addr 172.21.15.64:0/2385853627 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:18.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.990+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903040114 0 0) 0x7f08b00b40c0 con 0x7f08b0143d10 2024-03-29T10:49:18.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.990+0000 7f08aeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.990+0000 7f08aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.990+0000 7f08aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.990+0000 7f08aeffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.990+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0898003520 con 0x7f08b0143d10 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894156869 0 0) 0x7f0898003520 con 0x7f08b0143d10 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 >> v1:172.21.15.174:6789/0 conn(0x7f08b0071eb0 legacy=0x7f08b0146470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 >> v1:172.21.15.64:6789/0 conn(0x7f08b00b1f60 legacy=0x7f08b0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 monclient: found mon.noname-b 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b0155d70 con 0x7f08b0143d10 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:18.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08b6431640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08b6431640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f08a8001a10 con 0x7f08b0143d10 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 10 monclient: dump: 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:18.977 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08b6431640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.991+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08a8001ca0 con 0x7f08b0143d10 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.992+0000 7f08aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.992+0000 7f08b6431640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.992+0000 7f08b6431640 4 set_mon_vals no callback set 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08aeffd640 1 -- 172.21.15.64:0/2385853627 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f08a8002030 con 0x7f08b0143d10 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08aeffd640 10 monclient: dump: 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.978 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 10 monclient: shutdown 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 1 -- 172.21.15.64:0/2385853627 >> v1:172.21.15.64:6790/0 conn(0x7f08b0143d10 legacy=0x7f08b0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 1 -- 172.21.15.64:0/2385853627 shutdown_connections 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 1 -- 172.21.15.64:0/2385853627 >> 172.21.15.64:0/2385853627 conn(0x7f08b0066a00 msgr2=0x7f08b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 1 -- 172.21.15.64:0/2385853627 shutdown_connections 2024-03-29T10:49:18.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.993+0000 7f08b6431640 1 -- 172.21.15.64:0/2385853627 wait complete. 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.994+0000 7f08b6431640 10 monclient: build_initial_monmap 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.994+0000 7f08b6431640 10 monclient: monmap: 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:18.990 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 1 librados: starting msgr at 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 1 librados: starting objecter 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 1 Processor -- start 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 1 -- start start 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 1 librados: setting wanted keys 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 1 librados: calling monclient init 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.995+0000 7f08b6431640 10 monclient: init 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient(hunting): picked mon.noname-b con 0x7f08b00b1f60 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient(hunting): picked mon.noname-a con 0x7f08b0143d10 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient(hunting): picked mon.noname-c con 0x7f08b0149610 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b0149b30 con 0x7f08b0143d10 2024-03-29T10:49:18.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b013bb90 con 0x7f08b00b1f60 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b014af60 con 0x7f08b0149610 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.996+0000 7f08b6431640 10 monclient(hunting): authenticate will time out at 970.295238s 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.998+0000 7f08af7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f08b0143d10 0x7f08b013ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45958/0 (socket says 172.21.15.64:45958) 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.998+0000 7f08af7fe640 1 -- 172.21.15.64:0/2208285316 learned_addr learned my addr 172.21.15.64:0/2208285316 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 client.?.objecter ms_handle_connect 0x7f08b00b1f60 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 client.?.objecter ms_handle_connect 0x7f08b0149610 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 client.?.objecter ms_handle_connect 0x7f08b0143d10 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327784930 0 0) 0x7f08b0149b30 con 0x7f08b0143d10 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:18.999+0000 7f08acff9640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f089c004eb0 con 0x7f08b0143d10 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843819499 0 0) 0x7f08b013bb90 con 0x7f08b00b1f60 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b0149b30 con 0x7f08b00b1f60 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840452518 0 0) 0x7f08b014af60 con 0x7f08b0149610 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:49:18.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b013bb90 con 0x7f08b0149610 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 566835690 0 0) 0x7f089c004eb0 con 0x7f08b0143d10 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08b014af60 con 0x7f08b0143d10 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f08a4001870 con 0x7f08b0143d10 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): dump: 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.994 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189593660 0 0) 0x7f08b0149b30 con 0x7f08b00b1f60 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f089c004eb0 con 0x7f08b00b1f60 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3355030517 0 0) 0x7f08b014af60 con 0x7f08b0143d10 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 >> v1:172.21.15.64:6790/0 conn(0x7f08b0149610 legacy=0x7f08b013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.000+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 >> v1:172.21.15.174:6789/0 conn(0x7f08b00b1f60 legacy=0x7f08b0147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 1 monclient: found mon.a 2024-03-29T10:49:18.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b0147ea0 con 0x7f08b0143d10 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 10 monclient: _renew_subs 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b0148090 con 0x7f08b0143d10 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 10 monclient: _renew_subs 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b014a200 con 0x7f08b0143d10 2024-03-29T10:49:18.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08b6431640 1 librados: init done 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08a40019d0 con 0x7f08b0143d10 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.001+0000 7f08ae7fc640 4 set_mon_vals no callback set 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.003+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f08a4004aa0 con 0x7f08b0143d10 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.003+0000 7f08acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.003+0000 7f08acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.003+0000 7f08acff9640 10 monclient: dump: 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:18.997 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.003+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f08a401aa70 con 0x7f08b0143d10 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.004+0000 7f08affff640 10 monclient: get_auth_request con 0x7f089c06dd70 auth_method 0 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.004+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f08a40849d0 con 0x7f08b0143d10 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.004+0000 7f08acff9640 10 client.4332.objecter ms_dispatch 0x7f08b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.004+0000 7f08acff9640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.004+0000 7f08acff9640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.004+0000 7f08acff9640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.005+0000 7f08acff9640 10 client.4332.objecter ms_handle_connect 0x7f089c06dd70 2024-03-29T10:49:18.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.006+0000 7f08b6431640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:18.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.006+0000 7f08b6431640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:18.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.006+0000 7f08b6431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:18.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.006+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0890005080 con 0x7f08b0143d10 2024-03-29T10:49:18.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.015+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08a4050370 con 0x7f08b0143d10 2024-03-29T10:49:18.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.015+0000 7f08acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:19.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.015+0000 7f08acff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:19.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.361+0000 7f08b6431640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:19.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.361+0000 7f08b6431640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:19.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.361+0000 7f08b6431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:19.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.361+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0890005330 con 0x7f08b0143d10 2024-03-29T10:49:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.363+0000 7f08acff9640 1 -- 172.21.15.64:0/2208285316 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f08a40522a0 con 0x7f08b0143d10 2024-03-29T10:49:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.364+0000 7f08acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:19.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.364+0000 7f08acff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:19.346 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:19.346 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.368+0000 7f08b6431640 10 librados: watch_flush enter 2024-03-29T10:49:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 10 librados: watch_flush exit 2024-03-29T10:49:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T10:49:19.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 10 client.4332.objecter successfully canceled tick 2024-03-29T10:49:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 >> v1:172.21.15.174:6816/2473791532 conn(0x7f089c06dd70 legacy=0x7f089c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 10 monclient: shutdown 2024-03-29T10:49:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 >> v1:172.21.15.64:6789/0 conn(0x7f08b0143d10 legacy=0x7f08b013ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 shutdown_connections 2024-03-29T10:49:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 >> 172.21.15.64:0/2208285316 conn(0x7f08b0066a00 msgr2=0x7f08b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 shutdown_connections 2024-03-29T10:49:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.371+0000 7f08b6431640 1 -- 172.21.15.64:0/2208285316 wait complete. 2024-03-29T10:49:19.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:19.372+0000 7f08b6431640 1 librados: shutdown 2024-03-29T10:49:19.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:26.181 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:26.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.347+0000 7ff6785e8640 10 monclient: get_monmap_and_config 2024-03-29T10:49:26.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.347+0000 7ff6785e8640 10 monclient: build_initial_monmap 2024-03-29T10:49:26.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.347+0000 7ff6785e8640 10 monclient: monmap: 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:26.348647+0000 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:26.348647+0000 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:26.330 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:26.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.348+0000 7ff6785e8640 1 Processor -- start 2024-03-29T10:49:26.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.348+0000 7ff6785e8640 1 -- start start 2024-03-29T10:49:26.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.348+0000 7ff6785e8640 10 monclient: init 2024-03-29T10:49:26.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:26.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff6700ae9b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff670143f50 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6700b4890 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700aed90 con 0x7ff6700ae9b0 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700af020 con 0x7ff6700b4890 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700af300 con 0x7ff670143f50 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff676b85640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff6700b4890 0x7ff670151df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:48586/0 (socket says 172.21.15.64:48586) 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff676b85640 1 -- 172.21.15.64:0/3524303994 learned_addr learned my addr 172.21.15.64:0/3524303994 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.349+0000 7ff6785e8640 10 monclient(hunting): authenticate will time out at 977.648847s 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682942533 0 0) 0x7ff6700aed90 con 0x7ff6700ae9b0 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff675382640 10 monclient(hunting): _negotiate 2024-03-29T10:49:26.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff675382640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff675382640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff675382640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.350+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c003520 con 0x7ff6700ae9b0 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238851634 0 0) 0x7ff6700af020 con 0x7ff6700b4890 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient(hunting): _negotiate 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:49:26.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6700aed90 con 0x7ff6700b4890 2024-03-29T10:49:26.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 40447436 0 0) 0x7ff65c003520 con 0x7ff6700ae9b0 2024-03-29T10:49:26.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 >> v1:172.21.15.64:6790/0 conn(0x7ff670143f50 legacy=0x7ff6700b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 >> v1:172.21.15.174:6789/0 conn(0x7ff6700b4890 legacy=0x7ff670151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: _finish_hunting 0 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 monclient: found mon.noname-a 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700b0710 con 0x7ff6700ae9b0 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: _finish_auth 0 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: _check_auth_tickets 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff668001a10 con 0x7ff6700ae9b0 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:26.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff675382640 10 monclient: dump: 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff6785e8640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.351+0000 7ff6785e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff668001b70 con 0x7ff6700ae9b0 2024-03-29T10:49:26.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff675382640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff675382640 1 -- 172.21.15.64:0/3524303994 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff668001f00 con 0x7ff6700ae9b0 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff675382640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff675382640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff675382640 10 monclient: dump: 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:26.338 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff6785e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.352+0000 7ff6785e8640 4 set_mon_vals no callback set 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 10 monclient: shutdown 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 1 -- 172.21.15.64:0/3524303994 >> v1:172.21.15.64:6789/0 conn(0x7ff6700ae9b0 legacy=0x7ff670143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 1 -- 172.21.15.64:0/3524303994 shutdown_connections 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 1 -- 172.21.15.64:0/3524303994 >> 172.21.15.64:0/3524303994 conn(0x7ff670066a00 msgr2=0x7ff670066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 1 -- 172.21.15.64:0/3524303994 shutdown_connections 2024-03-29T10:49:26.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.353+0000 7ff6785e8640 1 -- 172.21.15.64:0/3524303994 wait complete. 2024-03-29T10:49:26.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.354+0000 7ff6785e8640 10 monclient: build_initial_monmap 2024-03-29T10:49:26.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.354+0000 7ff6785e8640 10 monclient: monmap: 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 1 librados: starting msgr at 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 1 librados: starting objecter 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 1 Processor -- start 2024-03-29T10:49:26.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 1 -- start start 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 1 librados: setting wanted keys 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 1 librados: calling monclient init 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.355+0000 7ff6785e8640 10 monclient: init 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff6700ae9b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6700b4890 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff670143f50 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700ad690 con 0x7ff670143f50 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700ac690 con 0x7ff6700ae9b0 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700aa0d0 con 0x7ff6700b4890 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:26.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff6785e8640 10 monclient(hunting): authenticate will time out at 977.655128s 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff676384640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff6700ae9b0 0x7ff6700ace00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:48828/0 (socket says 172.21.15.64:48828) 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff676384640 1 -- 172.21.15.64:0/1262621400 learned_addr learned my addr 172.21.15.64:0/1262621400 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 client.?.objecter ms_handle_connect 0x7ff6700ae9b0 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852113649 0 0) 0x7ff6700ac690 con 0x7ff6700ae9b0 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff654004eb0 con 0x7ff6700ae9b0 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 client.?.objecter ms_handle_connect 0x7ff670143f50 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.356+0000 7ff666ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:26.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880018470 0 0) 0x7ff654004eb0 con 0x7ff6700ae9b0 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6700ac690 con 0x7ff6700ae9b0 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff668001e80 con 0x7ff6700ae9b0 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:26.347 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 client.?.objecter ms_handle_connect 0x7ff6700b4890 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894493137 0 0) 0x7ff6700aa0d0 con 0x7ff6700b4890 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff654004eb0 con 0x7ff6700b4890 2024-03-29T10:49:26.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 158176616 0 0) 0x7ff6700ac690 con 0x7ff6700ae9b0 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 >> v1:172.21.15.64:6790/0 conn(0x7ff6700b4890 legacy=0x7ff6700aa7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 >> v1:172.21.15.64:6789/0 conn(0x7ff670143f50 legacy=0x7ff6700aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 monclient: found mon.b 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700aa260 con 0x7ff6700ae9b0 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff666ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 10 monclient: _renew_subs 2024-03-29T10:49:26.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6700aa4d0 con 0x7ff6700ae9b0 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 10 monclient: _renew_subs 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.357+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6701e7430 con 0x7ff6700ae9b0 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff6785e8640 1 librados: init done 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff668003c20 con 0x7ff6700ae9b0 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff666ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff668004030 con 0x7ff6700ae9b0 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff666ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff666ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff666ffd640 10 monclient: dump: 2024-03-29T10:49:26.350 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.358+0000 7ff674b81640 4 set_mon_vals no callback set 2024-03-29T10:49:26.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.360+0000 7ff6785e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.360+0000 7ff6785e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.360+0000 7ff6785e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.360+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff658005080 con 0x7ff6700ae9b0 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.360+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7ff668004190 con 0x7ff6700ae9b0 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7ff668084f20 con 0x7ff6700ae9b0 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff666ffd640 10 client.4360.objecter ms_dispatch 0x7ff670000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff666ffd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff666ffd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff666ffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff675b83640 10 monclient: get_auth_request con 0x7ff65406dce0 auth_method 0 2024-03-29T10:49:26.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.361+0000 7ff666ffd640 10 client.4360.objecter ms_handle_connect 0x7ff65406dce0 2024-03-29T10:49:26.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.367+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff668050800 con 0x7ff6700ae9b0 2024-03-29T10:49:26.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.367+0000 7ff666ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:26.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.367+0000 7ff666ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:26.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.769+0000 7ff6785e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:26.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.769+0000 7ff6785e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:26.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.769+0000 7ff6785e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:26.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.769+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff658005330 con 0x7ff6700ae9b0 2024-03-29T10:49:26.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.770+0000 7ff666ffd640 1 -- 172.21.15.64:0/1262621400 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7ff668052730 con 0x7ff6700ae9b0 2024-03-29T10:49:26.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.770+0000 7ff666ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:26.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.770+0000 7ff666ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:26.753 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:26.753 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:26.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.773+0000 7ff6785e8640 10 librados: watch_flush enter 2024-03-29T10:49:26.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.773+0000 7ff6785e8640 10 librados: watch_flush exit 2024-03-29T10:49:26.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.773+0000 7ff6785e8640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T10:49:26.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.773+0000 7ff6785e8640 10 client.4360.objecter successfully canceled tick 2024-03-29T10:49:26.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 >> v1:172.21.15.174:6816/2473791532 conn(0x7ff65406dce0 legacy=0x7ff6540701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 10 monclient: shutdown 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 >> v1:172.21.15.174:6789/0 conn(0x7ff6700ae9b0 legacy=0x7ff6700ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 shutdown_connections 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 >> 172.21.15.64:0/1262621400 conn(0x7ff670066a00 msgr2=0x7ff6700b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 shutdown_connections 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 -- 172.21.15.64:0/1262621400 wait complete. 2024-03-29T10:49:26.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:26.774+0000 7ff6785e8640 1 librados: shutdown 2024-03-29T10:49:26.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:33.578 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.739+0000 7f5cf09a4640 10 monclient: get_monmap_and_config 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.739+0000 7f5cf09a4640 10 monclient: build_initial_monmap 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.739+0000 7f5cf09a4640 10 monclient: monmap: 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:33.741691+0000 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:33.741691+0000 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:33.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:33.724 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:33.724 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.740+0000 7f5cf09a4640 1 Processor -- start 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.740+0000 7f5cf09a4640 1 -- start start 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.740+0000 7f5cf09a4640 10 monclient: init 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5cec0b4330 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:33.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 10 monclient(hunting): picked mon.noname-c con 0x7f5cec1470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5cec1439b0 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec06f2a0 con 0x7f5cec0b4330 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec146b90 con 0x7f5cec1470b0 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0b2ca0 con 0x7f5cec1439b0 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.741+0000 7f5cf09a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5cf09a4640 10 monclient(hunting): authenticate will time out at 985.041860s 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5cea59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5cec0b4330 0x7f5cec0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:53064/0 (socket says 172.21.15.64:53064) 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5cea59c640 1 -- 172.21.15.64:0/1516632863 learned_addr learned my addr 172.21.15.64:0/1516632863 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214610947 0 0) 0x7f5cec06f2a0 con 0x7f5cec0b4330 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5ce959a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5ce959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5ce959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:33.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5ce959a640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:49:33.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.742+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd8003520 con 0x7f5cec0b4330 2024-03-29T10:49:33.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.743+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3483498478 0 0) 0x7f5cd8003520 con 0x7f5cec0b4330 2024-03-29T10:49:33.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.743+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 >> v1:172.21.15.64:6790/0 conn(0x7f5cec1439b0 legacy=0x7f5cec151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:33.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 >> v1:172.21.15.174:6789/0 conn(0x7f5cec1470b0 legacy=0x7f5cec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 1 monclient: found mon.noname-a 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec06e0c0 con 0x7f5cec0b4330 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: _finish_auth 0 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5cd4001690 con 0x7f5cec0b4330 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: dump: 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:33.729 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cd4001860 con 0x7f5cec0b4330 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5cf09a4640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5cf09a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5ce959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:33.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5cf09a4640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.745+0000 7f5cf09a4640 4 set_mon_vals no callback set 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5ce959a640 1 -- 172.21.15.64:0/1516632863 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5cd4001c70 con 0x7f5cec0b4330 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5ce959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5ce959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5ce959a640 10 monclient: dump: 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:33.731 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:33.732 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:33.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5cf09a4640 10 monclient: shutdown 2024-03-29T10:49:33.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5cf09a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:33.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1516632863 >> v1:172.21.15.64:6789/0 conn(0x7f5cec0b4330 legacy=0x7f5cec0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:33.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1516632863 shutdown_connections 2024-03-29T10:49:33.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.746+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1516632863 >> 172.21.15.64:0/1516632863 conn(0x7f5cec066a00 msgr2=0x7f5cec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:33.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.747+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1516632863 shutdown_connections 2024-03-29T10:49:33.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.747+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1516632863 wait complete. 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.748+0000 7f5cf09a4640 10 monclient: build_initial_monmap 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.748+0000 7f5cf09a4640 10 monclient: monmap: 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:33.735 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.748+0000 7f5cf09a4640 1 librados: starting msgr at 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.748+0000 7f5cf09a4640 1 librados: starting objecter 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.748+0000 7f5cf09a4640 1 Processor -- start 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 1 -- start start 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 1 librados: setting wanted keys 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 1 librados: calling monclient init 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient: init 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient(hunting): picked mon.noname-c con 0x7f5cec0b4330 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5cec1439b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5cec1470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:33.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec122050 con 0x7f5cec1439b0 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec1222c0 con 0x7f5cec1470b0 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec120fe0 con 0x7f5cec0b4330 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.749+0000 7f5cf09a4640 10 monclient(hunting): authenticate will time out at 985.049797s 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.750+0000 7f5cead9d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f5cec1470b0 0x7f5cec120370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:50536/0 (socket says 172.21.15.64:50536) 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.750+0000 7f5cead9d640 1 -- 172.21.15.64:0/1124174280 learned_addr learned my addr 172.21.15.64:0/1124174280 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.750+0000 7f5ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cec1470b0 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.750+0000 7f5ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.751+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023640911 0 0) 0x7f5cec1222c0 con 0x7f5cec1470b0 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.752+0000 7f5ce2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.752+0000 7f5ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.752+0000 7f5ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:33.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.752+0000 7f5ce2ffd640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:49:33.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.752+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc004eb0 con 0x7f5cec1470b0 2024-03-29T10:49:33.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cec1439b0 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254968986 0 0) 0x7f5cec122050 con 0x7f5cec1439b0 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec1222c0 con 0x7f5cec1439b0 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703553955 0 0) 0x7f5cec1222c0 con 0x7f5cec1439b0 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cec122050 con 0x7f5cec1439b0 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5cdc001c30 con 0x7f5cec1439b0 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:33.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.753+0000 7f5ce2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3945803635 0 0) 0x7f5cec122050 con 0x7f5cec1439b0 2024-03-29T10:49:33.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 >> v1:172.21.15.64:6790/0 conn(0x7f5cec0b4330 legacy=0x7f5cec1df150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 >> v1:172.21.15.174:6789/0 conn(0x7f5cec1470b0 legacy=0x7f5cec120370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 1 monclient: found mon.a 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec120a80 con 0x7f5cec1439b0 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cec0b4330 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:49:33.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 10 monclient: _renew_subs 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec120cd0 con 0x7f5cec1439b0 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 10 monclient: _renew_subs 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec142ce0 con 0x7f5cec1439b0 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.754+0000 7f5cf09a4640 1 librados: init done 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.755+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cdc003990 con 0x7f5cec1439b0 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.755+0000 7f5ce2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.755+0000 7f5ce8d99640 4 set_mon_vals no callback set 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5cdc003ca0 con 0x7f5cec1439b0 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:33.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cec121220 con 0x7f5cec1439b0 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.756+0000 7f5ce2ffd640 10 monclient: dump: 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:33.748 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.757+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f5cdc019c70 con 0x7f5cec1439b0 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f5cdc083c40 con 0x7f5cec1439b0 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5ce2ffd640 10 client.4344.objecter ms_dispatch 0x7f5cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5ce2ffd640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5ce2ffd640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5ce2ffd640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5cea59c640 10 monclient: get_auth_request con 0x7f5ccc06dd00 auth_method 0 2024-03-29T10:49:33.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.758+0000 7f5ce2ffd640 10 client.4344.objecter ms_handle_connect 0x7f5ccc06dd00 2024-03-29T10:49:33.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.765+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5cec121220 con 0x7f5cec1439b0 2024-03-29T10:49:33.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.765+0000 7f5ce2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:33.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:33.765+0000 7f5ce2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:34.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.145+0000 7f5ce0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:34.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.145+0000 7f5ce0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:34.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.145+0000 7f5ce0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:49:34.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.145+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cec121220 con 0x7f5cec1439b0 2024-03-29T10:49:34.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.187+0000 7f5ce2ffd640 1 -- 172.21.15.64:0/1124174280 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f5cec121220 con 0x7f5cec1439b0 2024-03-29T10:49:34.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.187+0000 7f5ce2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:34.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.187+0000 7f5ce2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:34.171 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:34.171 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:34.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.192+0000 7f5ce0ff9640 10 librados: watch_flush enter 2024-03-29T10:49:34.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.192+0000 7f5ce0ff9640 10 librados: watch_flush exit 2024-03-29T10:49:34.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.192+0000 7f5ce0ff9640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.192+0000 7f5ce0ff9640 10 client.4344.objecter successfully canceled tick 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 >> v1:172.21.15.174:6816/2473791532 conn(0x7f5ccc06dd00 legacy=0x7f5ccc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 10 monclient: shutdown 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 >> v1:172.21.15.64:6789/0 conn(0x7f5cec1439b0 legacy=0x7f5cec11fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 shutdown_connections 2024-03-29T10:49:34.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 >> 172.21.15.64:0/1124174280 conn(0x7f5cec066a00 msgr2=0x7f5cec1ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:34.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 shutdown_connections 2024-03-29T10:49:34.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 -- 172.21.15.64:0/1124174280 wait complete. 2024-03-29T10:49:34.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:34.193+0000 7f5ce0ff9640 1 librados: shutdown 2024-03-29T10:49:34.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:40.997 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:41.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.163+0000 7f025f700640 10 monclient: get_monmap_and_config 2024-03-29T10:49:41.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.163+0000 7f025f700640 10 monclient: build_initial_monmap 2024-03-29T10:49:41.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.163+0000 7f025f700640 10 monclient: monmap: 2024-03-29T10:49:41.147 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:41.147 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:41.147 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:41.164411+0000 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:41.164411+0000 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:41.148 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.166+0000 7f025f700640 1 Processor -- start 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.166+0000 7f025f700640 1 -- start start 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.166+0000 7f025f700640 10 monclient: init 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient(hunting): picked mon.noname-a con 0x7f02580b4100 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient(hunting): picked mon.noname-c con 0x7f025813ee90 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient(hunting): picked mon.noname-b con 0x7f02580af900 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580afd10 con 0x7f02580b4100 2024-03-29T10:49:41.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580af6f0 con 0x7f025813ee90 2024-03-29T10:49:41.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580aaf60 con 0x7f02580af900 2024-03-29T10:49:41.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:41.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025f700640 10 monclient(hunting): authenticate will time out at 992.466569s 2024-03-29T10:49:41.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025cc9b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f025813ee90 0x7f025813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:47066/0 (socket says 172.21.15.64:47066) 2024-03-29T10:49:41.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.167+0000 7f025cc9b640 1 -- 172.21.15.64:0/1077400352 learned_addr learned my addr 172.21.15.64:0/1077400352 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:41.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731397137 0 0) 0x7f02580af6f0 con 0x7f025813ee90 2024-03-29T10:49:41.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:41.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:41.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:41.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0244003520 con 0x7f025813ee90 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141608155 0 0) 0x7f02580afd10 con 0x7f02580b4100 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:49:41.153 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02580af6f0 con 0x7f02580b4100 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236339287 0 0) 0x7f0244003520 con 0x7f025813ee90 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 >> v1:172.21.15.64:6790/0 conn(0x7f02580af900 legacy=0x7f0258149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 >> v1:172.21.15.64:6789/0 conn(0x7f02580b4100 legacy=0x7f02580a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 monclient: found mon.noname-c 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025813e970 con 0x7f025813ee90 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: _finish_auth 0 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0250001a10 con 0x7f025813ee90 2024-03-29T10:49:41.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f024ffff640 10 monclient: dump: 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:41.155 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f025f700640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.171+0000 7f025f700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.172+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0250001b70 con 0x7f025813ee90 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.173+0000 7f024ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.173+0000 7f024ffff640 1 -- 172.21.15.64:0/1077400352 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0250001f80 con 0x7f025813ee90 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.173+0000 7f024ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.173+0000 7f024ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.173+0000 7f024ffff640 10 monclient: dump: 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:41.156 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:41.157 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:41.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.174+0000 7f025f700640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:41.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.174+0000 7f025f700640 4 set_mon_vals no callback set 2024-03-29T10:49:41.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 10 monclient: shutdown 2024-03-29T10:49:41.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:41.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 1 -- 172.21.15.64:0/1077400352 >> v1:172.21.15.174:6789/0 conn(0x7f025813ee90 legacy=0x7f025813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 1 -- 172.21.15.64:0/1077400352 shutdown_connections 2024-03-29T10:49:41.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 1 -- 172.21.15.64:0/1077400352 >> 172.21.15.64:0/1077400352 conn(0x7f02580b35e0 msgr2=0x7f02580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:41.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 1 -- 172.21.15.64:0/1077400352 shutdown_connections 2024-03-29T10:49:41.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.176+0000 7f025f700640 1 -- 172.21.15.64:0/1077400352 wait complete. 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.177+0000 7f025f700640 10 monclient: build_initial_monmap 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.177+0000 7f025f700640 10 monclient: monmap: 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:41.160 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.177+0000 7f025f700640 1 librados: starting msgr at 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.177+0000 7f025f700640 1 librados: starting objecter 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.178+0000 7f025f700640 1 Processor -- start 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.178+0000 7f025f700640 1 -- start start 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.178+0000 7f025f700640 1 librados: setting wanted keys 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.178+0000 7f025f700640 1 librados: calling monclient init 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.178+0000 7f025f700640 10 monclient: init 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.179+0000 7f025f700640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:41.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.179+0000 7f025f700640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.179+0000 7f025f700640 10 monclient(hunting): picked mon.noname-a con 0x7f02580af900 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.179+0000 7f025f700640 10 monclient(hunting): picked mon.noname-b con 0x7f02580b4100 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.179+0000 7f025f700640 10 monclient(hunting): picked mon.noname-c con 0x7f025813ee90 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.180+0000 7f025f700640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025811cb60 con 0x7f02580af900 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.180+0000 7f025f700640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025811b980 con 0x7f02580b4100 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.180+0000 7f025f700640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025811bb10 con 0x7f025813ee90 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.180+0000 7f025f700640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:41.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.180+0000 7f025f700640 10 monclient(hunting): authenticate will time out at 992.478886s 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f025cc9b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f02580b4100 0x7f025811e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:48102/0 (socket says 172.21.15.64:48102) 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f025cc9b640 1 -- 172.21.15.64:0/1277305178 learned_addr learned my addr 172.21.15.64:0/1277305178 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 10 client.?.objecter ms_handle_connect 0x7f02580b4100 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830828657 0 0) 0x7f025811b980 con 0x7f02580b4100 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:41.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.181+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023c004eb0 con 0x7f02580b4100 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061330432 0 0) 0x7f023c004eb0 con 0x7f02580b4100 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f025811b980 con 0x7f02580b4100 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0250001f40 con 0x7f02580b4100 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 10 monclient(hunting): dump: 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.169 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1354627967 0 0) 0x7f025811b980 con 0x7f02580b4100 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.182+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 >> v1:172.21.15.64:6790/0 conn(0x7f025813ee90 legacy=0x7f025811eb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 >> v1:172.21.15.64:6789/0 conn(0x7f02580af900 legacy=0x7f025811c390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 1 monclient: found mon.b 2024-03-29T10:49:41.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025811bca0 con 0x7f02580b4100 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f024dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f025f700640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f025f700640 10 monclient: _renew_subs 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f025f700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.184+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0258071c90 con 0x7f02580b4100 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.185+0000 7f025f700640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.185+0000 7f025f700640 10 monclient: _renew_subs 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.185+0000 7f025f700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.185+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02581efbb0 con 0x7f02580b4100 2024-03-29T10:49:41.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.185+0000 7f025f700640 1 librados: init done 2024-03-29T10:49:41.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.187+0000 7f025f700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.187+0000 7f025f700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.187+0000 7f025f700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.187+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025813f3b0 con 0x7f02580b4100 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0250004e10 con 0x7f02580b4100 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f02500020a0 con 0x7f02580b4100 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024dffb640 10 monclient: dump: 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:41.174 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.188+0000 7f024f7fe640 4 set_mon_vals no callback set 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.189+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f025001aeb0 con 0x7f02580b4100 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.190+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f0250084d60 con 0x7f02580b4100 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.190+0000 7f024dffb640 10 client.4372.objecter ms_dispatch 0x7f0258000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.190+0000 7f024dffb640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.190+0000 7f024dffb640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.190+0000 7f024dffb640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T10:49:41.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.190+0000 7f025d49c640 10 monclient: get_auth_request con 0x7f023c06dcd0 auth_method 0 2024-03-29T10:49:41.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.191+0000 7f024dffb640 10 client.4372.objecter ms_handle_connect 0x7f023c06dcd0 2024-03-29T10:49:41.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.196+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0250050470 con 0x7f02580b4100 2024-03-29T10:49:41.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.196+0000 7f024dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:41.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.196+0000 7f024dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:41.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.528+0000 7f025f700640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:41.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.528+0000 7f025f700640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:41.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.528+0000 7f025f700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:41.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.528+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f025811ccf0 con 0x7f02580b4100 2024-03-29T10:49:41.513 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:41.514 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:41.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.529+0000 7f024dffb640 1 -- 172.21.15.64:0/1277305178 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f0250052570 con 0x7f02580b4100 2024-03-29T10:49:41.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.529+0000 7f024dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.529+0000 7f024dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 10 librados: watch_flush enter 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 10 librados: watch_flush exit 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 10 client.4372.objecter successfully canceled tick 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 >> v1:172.21.15.174:6816/2473791532 conn(0x7f023c06dcd0 legacy=0x7f023c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 10 monclient: shutdown 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 >> v1:172.21.15.174:6789/0 conn(0x7f02580b4100 legacy=0x7f025811e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 shutdown_connections 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.532+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 >> 172.21.15.64:0/1277305178 conn(0x7f02580b35e0 msgr2=0x7f02580ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.533+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 shutdown_connections 2024-03-29T10:49:41.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.533+0000 7f025f700640 1 -- 172.21.15.64:0/1277305178 wait complete. 2024-03-29T10:49:41.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:41.533+0000 7f025f700640 1 librados: shutdown 2024-03-29T10:49:41.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:48.338 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:48.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.504+0000 7fc55359e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:48.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.504+0000 7fc55359e640 10 monclient: build_initial_monmap 2024-03-29T10:49:48.486 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.504+0000 7fc55359e640 10 monclient: monmap: 2024-03-29T10:49:48.486 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:48.486 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:48.505051+0000 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:48.505051+0000 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:48.487 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:48.488 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.505+0000 7fc55359e640 1 Processor -- start 2024-03-29T10:49:48.488 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.505+0000 7fc55359e640 1 -- start start 2024-03-29T10:49:48.488 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.505+0000 7fc55359e640 10 monclient: init 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc54c02b4c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc54c00b8a0 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc54c02a5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c02c5c0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c02c830 con 0x7fc54c02b4c0 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c02afa0 con 0x7fc54c02a5a0 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:48.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.507+0000 7fc55359e640 10 monclient(hunting): authenticate will time out at 999.806012s 2024-03-29T10:49:48.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.509+0000 7fc55259c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc54c02b4c0 0x7fc54c006890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:55134/0 (socket says 172.21.15.64:55134) 2024-03-29T10:49:48.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.509+0000 7fc55259c640 1 -- 172.21.15.64:0/2647673935 learned_addr learned my addr 172.21.15.64:0/2647673935 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.510+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009980763 0 0) 0x7fc54c02c830 con 0x7fc54c02b4c0 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.510+0000 7fc55159a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.510+0000 7fc55159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.510+0000 7fc55159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.510+0000 7fc55159a640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.510+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c003520 con 0x7fc54c02b4c0 2024-03-29T10:49:48.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840063736 0 0) 0x7fc54c02c5c0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:48.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:48.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:48.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:49:48.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c02c830 con 0x7fc54c00b8a0 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648436474 0 0) 0x7fc54c02afa0 con 0x7fc54c02a5a0 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c02c5c0 con 0x7fc54c02a5a0 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2341179794 0 0) 0x7fc54c02c830 con 0x7fc54c00b8a0 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 >> v1:172.21.15.64:6790/0 conn(0x7fc54c02a5a0 legacy=0x7fc54c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 >> v1:172.21.15.174:6789/0 conn(0x7fc54c02b4c0 legacy=0x7fc54c006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:48.494 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 monclient: found mon.noname-a 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c02aba0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient: _finish_auth 0 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.511+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc548001670 con 0x7fc54c00b8a0 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.512+0000 7fc55159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.512+0000 7fc55159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.512+0000 7fc55159a640 10 monclient: dump: 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.495 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.512+0000 7fc55359e640 5 monclient: authenticate success, global_id 4350 2024-03-29T10:49:48.496 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.512+0000 7fc55359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc548001890 con 0x7fc54c00b8a0 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55159a640 1 -- 172.21.15.64:0/2647673935 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc548001c20 con 0x7fc54c00b8a0 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55159a640 10 monclient: dump: 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:48.498 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55359e640 4 set_mon_vals no callback set 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.513+0000 7fc55359e640 10 monclient: shutdown 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.514+0000 7fc55359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.514+0000 7fc55359e640 1 -- 172.21.15.64:0/2647673935 >> v1:172.21.15.64:6789/0 conn(0x7fc54c00b8a0 legacy=0x7fc54c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.514+0000 7fc55359e640 1 -- 172.21.15.64:0/2647673935 shutdown_connections 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.514+0000 7fc55359e640 1 -- 172.21.15.64:0/2647673935 >> 172.21.15.64:0/2647673935 conn(0x7fc54c0058f0 msgr2=0x7fc54c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.514+0000 7fc55359e640 1 -- 172.21.15.64:0/2647673935 shutdown_connections 2024-03-29T10:49:48.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.515+0000 7fc55359e640 1 -- 172.21.15.64:0/2647673935 wait complete. 2024-03-29T10:49:48.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.516+0000 7fc55359e640 10 monclient: build_initial_monmap 2024-03-29T10:49:48.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.516+0000 7fc55359e640 10 monclient: monmap: 2024-03-29T10:49:48.500 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:48.500 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:48.500 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:48.500 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 1 librados: starting msgr at 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 1 librados: starting objecter 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 1 Processor -- start 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 1 -- start start 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 1 librados: setting wanted keys 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 1 librados: calling monclient init 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.517+0000 7fc55359e640 10 monclient: init 2024-03-29T10:49:48.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.518+0000 7fc55359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:48.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.518+0000 7fc55359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:48.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.520+0000 7fc55359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc54c00b8a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:48.505 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc54c02a5a0 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc54c02b4c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c4480 con 0x7fc54c02a5a0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c46f0 con 0x7fc54c02b4c0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c49d0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.522+0000 7fc55359e640 10 monclient(hunting): authenticate will time out at 999.821522s 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.523+0000 7fc55259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc54c00b8a0 0x7fc54c0d59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41005/0 (socket says 172.21.15.64:41005) 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.523+0000 7fc55259c640 1 -- 172.21.15.64:0/1989963018 learned_addr learned my addr 172.21.15.64:0/1989963018 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 client.?.objecter ms_handle_connect 0x7fc54c00b8a0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 client.?.objecter ms_handle_connect 0x7fc54c02b4c0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928260285 0 0) 0x7fc54c0c49d0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:48.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:48.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:48.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:49:48.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534004eb0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 client.?.objecter ms_handle_connect 0x7fc54c02a5a0 2024-03-29T10:49:48.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3351043536 0 0) 0x7fc534004eb0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.524+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc54c0c49d0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc544001950 con 0x7fc54c00b8a0 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:48.513 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893004942 0 0) 0x7fc54c0c46f0 con 0x7fc54c02b4c0 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534004eb0 con 0x7fc54c02b4c0 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706942774 0 0) 0x7fc54c0c4480 con 0x7fc54c02a5a0 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:48.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c0c46f0 con 0x7fc54c02a5a0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4174163192 0 0) 0x7fc54c0c49d0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 >> v1:172.21.15.174:6789/0 conn(0x7fc54c02b4c0 legacy=0x7fc54c167cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 >> v1:172.21.15.64:6789/0 conn(0x7fc54c02a5a0 legacy=0x7fc54c0ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 monclient: found mon.c 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0c2c00 con 0x7fc54c00b8a0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.525+0000 7fc542ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5440047e0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:48.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc544001ab0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc542ffd640 10 monclient: dump: 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:48.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 10 monclient: _renew_subs 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c0d60e0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 10 monclient: _renew_subs 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c0c3080 con 0x7fc54c00b8a0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.526+0000 7fc55359e640 1 librados: init done 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.527+0000 7fc550d99640 4 set_mon_vals no callback set 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc55359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fc544004bf0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc55359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc55359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc538005080 con 0x7fc54c00b8a0 2024-03-29T10:49:48.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc551d9b640 10 monclient: get_auth_request con 0x7fc53406db40 auth_method 0 2024-03-29T10:49:48.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.529+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fc544084af0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.530+0000 7fc542ffd640 10 client.4379.objecter ms_dispatch 0x7fc554000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:48.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.530+0000 7fc542ffd640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:48.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.530+0000 7fc542ffd640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:48.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.530+0000 7fc542ffd640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:49:48.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.530+0000 7fc542ffd640 10 client.4379.objecter ms_handle_connect 0x7fc53406db40 2024-03-29T10:49:48.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.538+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5440503d0 con 0x7fc54c00b8a0 2024-03-29T10:49:48.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.538+0000 7fc542ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:48.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.538+0000 7fc542ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:48.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.878+0000 7fc55359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:48.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.878+0000 7fc55359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:48.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.878+0000 7fc55359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:49:48.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.878+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc538005330 con 0x7fc54c00b8a0 2024-03-29T10:49:48.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.879+0000 7fc542ffd640 1 -- 172.21.15.64:0/1989963018 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fc544052300 con 0x7fc54c00b8a0 2024-03-29T10:49:48.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.879+0000 7fc542ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:48.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.879+0000 7fc542ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:48.862 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:48.862 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 10 librados: watch_flush enter 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 10 librados: watch_flush exit 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 >> v1:172.21.15.174:6816/2473791532 conn(0x7fc53406db40 legacy=0x7fc534070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 10 monclient: shutdown 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 >> v1:172.21.15.64:6790/0 conn(0x7fc54c00b8a0 legacy=0x7fc54c0d59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 shutdown_connections 2024-03-29T10:49:48.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.883+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 >> 172.21.15.64:0/1989963018 conn(0x7fc54c0058f0 msgr2=0x7fc54c033e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:48.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.884+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 shutdown_connections 2024-03-29T10:49:48.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.884+0000 7fc55359e640 1 -- 172.21.15.64:0/1989963018 wait complete. 2024-03-29T10:49:48.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:48.884+0000 7fc55359e640 1 librados: shutdown 2024-03-29T10:49:48.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:55.689 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.857+0000 7f3a20884640 10 monclient: get_monmap_and_config 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.857+0000 7f3a20884640 10 monclient: build_initial_monmap 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.857+0000 7f3a20884640 10 monclient: monmap: 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:49:55.858132+0000 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:49:55.858132+0000 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.839 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:55.840 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:49:55.840 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:49:55.840 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.858+0000 7f3a20884640 1 Processor -- start 2024-03-29T10:49:55.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.858+0000 7f3a20884640 1 -- start start 2024-03-29T10:49:55.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.858+0000 7f3a20884640 10 monclient: init 2024-03-29T10:49:55.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:55.842 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient(hunting): picked mon.noname-a con 0x7f3a1c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient(hunting): picked mon.noname-b con 0x7f3a1c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c0aaef0 con 0x7f3a1c13f650 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c145bd0 con 0x7f3a1c1457f0 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c145eb0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:55.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.859+0000 7f3a20884640 10 monclient(hunting): authenticate will time out at 1007.158212s 2024-03-29T10:49:55.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1ad9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3a1c0a9d70 0x7f3a1c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54059/0 (socket says 172.21.15.64:54059) 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1ad9d640 1 -- 172.21.15.64:0/3461381162 learned_addr learned my addr 172.21.15.64:0/3461381162 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709954014 0 0) 0x7f3a1c145eb0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a04003520 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158375205 0 0) 0x7f3a04003520 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 >> v1:172.21.15.174:6789/0 conn(0x7f3a1c1457f0 legacy=0x7f3a1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 >> v1:172.21.15.64:6789/0 conn(0x7f3a1c13f650 legacy=0x7f3a1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 monclient: found mon.noname-b 2024-03-29T10:49:55.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.866+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c146340 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: _finish_auth 0 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3a0c001690 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: dump: 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.853 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a0c001920 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a20884640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a20884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a20884640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 1 -- 172.21.15.64:0/3461381162 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3a0c001cf0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a20884640 4 set_mon_vals no callback set 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.867+0000 7f3a1959a640 10 monclient: dump: 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.868+0000 7f3a20884640 10 monclient: shutdown 2024-03-29T10:49:55.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.868+0000 7f3a20884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:55.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.868+0000 7f3a20884640 1 -- 172.21.15.64:0/3461381162 >> v1:172.21.15.64:6790/0 conn(0x7f3a1c0a9d70 legacy=0x7f3a1c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:55.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.869+0000 7f3a20884640 1 -- 172.21.15.64:0/3461381162 shutdown_connections 2024-03-29T10:49:55.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.869+0000 7f3a20884640 1 -- 172.21.15.64:0/3461381162 >> 172.21.15.64:0/3461381162 conn(0x7f3a1c066a00 msgr2=0x7f3a1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:55.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.872+0000 7f3a20884640 1 -- 172.21.15.64:0/3461381162 shutdown_connections 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.872+0000 7f3a20884640 1 -- 172.21.15.64:0/3461381162 wait complete. 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.873+0000 7f3a20884640 10 monclient: build_initial_monmap 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.873+0000 7f3a20884640 10 monclient: monmap: 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:55.857 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 1 librados: starting msgr at 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 1 librados: starting objecter 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 1 Processor -- start 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 1 -- start start 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 1 librados: setting wanted keys 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 1 librados: calling monclient init 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.874+0000 7f3a20884640 10 monclient: init 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:55.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient(hunting): picked mon.noname-b con 0x7f3a1c0a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient(hunting): picked mon.noname-a con 0x7f3a1c1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c142640 con 0x7f3a1c1457f0 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c145340 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c1383c0 con 0x7f3a1c13f650 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:55.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.875+0000 7f3a20884640 10 monclient(hunting): authenticate will time out at 1007.174359s 2024-03-29T10:49:55.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a19d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3a1c13f650 0x7f3a1c139b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54171/0 (socket says 172.21.15.64:54171) 2024-03-29T10:49:55.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a19d9b640 1 -- 172.21.15.64:0/3380899454 learned_addr learned my addr 172.21.15.64:0/3380899454 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:49:55.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 client.?.objecter ms_handle_connect 0x7f3a1c13f650 2024-03-29T10:49:55.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:55.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 client.?.objecter ms_handle_connect 0x7f3a1c1457f0 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 client.?.objecter ms_handle_connect 0x7f3a1c0a9d70 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899773749 0 0) 0x7f3a1c1383c0 con 0x7f3a1c13f650 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:55.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39fc004eb0 con 0x7f3a1c13f650 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631696164 0 0) 0x7f3a1c145340 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a1c1383c0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271157124 0 0) 0x7f3a1c142640 con 0x7f3a1c1457f0 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.882+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a1c145340 con 0x7f3a1c1457f0 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820247111 0 0) 0x7f3a1c1383c0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a1c142640 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3a14001af0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 10 monclient(hunting): dump: 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.884+0000 7f3a0affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.885+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657729135 0 0) 0x7f3a1c145340 con 0x7f3a1c1457f0 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.885+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39fc005b20 con 0x7f3a1c1457f0 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.885+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3a0c001550 con 0x7f3a1c1457f0 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.885+0000 7f3a0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.885+0000 7f3a0affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.885+0000 7f3a0affd640 10 monclient(hunting): dump: 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.869 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 161441202 0 0) 0x7f3a1c142640 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 >> v1:172.21.15.64:6790/0 conn(0x7f3a1c13f650 legacy=0x7f3a1c139b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:55.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 >> v1:172.21.15.64:6789/0 conn(0x7f3a1c1457f0 legacy=0x7f3a1c137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 1 monclient: found mon.b 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c142d50 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 10 monclient: _finish_auth 0 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 5 monclient: authenticate success, global_id 4381 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 10 monclient: _renew_subs 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c144d30 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:55.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 10 monclient: _renew_subs 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c200ba0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.886+0000 7f3a20884640 1 librados: init done 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a14001fe0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3a14001af0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a0affd640 10 monclient: dump: 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:55.872 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a18d99640 4 set_mon_vals no callback set 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.887+0000 7f3a20884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.888+0000 7f3a20884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.888+0000 7f3a20884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:55.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.888+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a00005080 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.889+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f3a140197e0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.890+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f3a140534d0 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.890+0000 7f3a0affd640 10 client.4381.objecter ms_dispatch 0x7f3a1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:55.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.890+0000 7f3a0affd640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:55.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.890+0000 7f3a0affd640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:55.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.890+0000 7f3a0affd640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T10:49:55.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.890+0000 7f3a19d9b640 10 monclient: get_auth_request con 0x7f39fc06e2f0 auth_method 0 2024-03-29T10:49:55.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.891+0000 7f3a0affd640 10 client.4381.objecter ms_handle_connect 0x7f39fc06e2f0 2024-03-29T10:49:55.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.897+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a1404f770 con 0x7f3a1c0a9d70 2024-03-29T10:49:55.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.897+0000 7f3a0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:55.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:55.897+0000 7f3a0affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:56.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.222+0000 7f3a20884640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:56.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.222+0000 7f3a20884640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:56.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.222+0000 7f3a20884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:49:56.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.222+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a00005330 con 0x7f3a1c0a9d70 2024-03-29T10:49:56.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.225+0000 7f3a0affd640 1 -- 172.21.15.64:0/3380899454 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f3a1404f910 con 0x7f3a1c0a9d70 2024-03-29T10:49:56.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.225+0000 7f3a0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:56.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.225+0000 7f3a0affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:56.211 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:49:56.211 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:56.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.231+0000 7f3a20884640 10 librados: watch_flush enter 2024-03-29T10:49:56.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.233+0000 7f3a20884640 10 librados: watch_flush exit 2024-03-29T10:49:56.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.233+0000 7f3a20884640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T10:49:56.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.233+0000 7f3a20884640 10 client.4381.objecter successfully canceled tick 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 >> v1:172.21.15.174:6816/2473791532 conn(0x7f39fc06e2f0 legacy=0x7f39fc0707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 10 monclient: shutdown 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 >> v1:172.21.15.174:6789/0 conn(0x7f3a1c0a9d70 legacy=0x7f3a1c1d97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 shutdown_connections 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 >> 172.21.15.64:0/3380899454 conn(0x7f3a1c066a00 msgr2=0x7f3a1c1da870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 shutdown_connections 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 -- 172.21.15.64:0/3380899454 wait complete. 2024-03-29T10:49:56.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:49:56.234+0000 7f3a20884640 1 librados: shutdown 2024-03-29T10:49:56.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:03.042 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.210+0000 7f5b63f6d640 10 monclient: get_monmap_and_config 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.210+0000 7f5b63f6d640 10 monclient: build_initial_monmap 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.210+0000 7f5b63f6d640 10 monclient: monmap: 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:03.211186+0000 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:03.211186+0000 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:03.192 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.193 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:03.193 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:03.193 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:03.193 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.213+0000 7f5b63f6d640 1 Processor -- start 2024-03-29T10:50:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.214+0000 7f5b63f6d640 1 -- start start 2024-03-29T10:50:03.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.214+0000 7f5b63f6d640 10 monclient: init 2024-03-29T10:50:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.215+0000 7f5b63f6d640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.215+0000 7f5b63f6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:03.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.215+0000 7f5b63f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b5c1422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b61508640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5b5c142690 0x7f5b5c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:55833/0 (socket says 172.21.15.64:55833) 2024-03-29T10:50:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b5c142690 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b5c0b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c142c80 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0aae40 con 0x7f5b5c1422b0 2024-03-29T10:50:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c13fbd0 con 0x7f5b5c142690 2024-03-29T10:50:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.217+0000 7f5b63f6d640 10 monclient(hunting): authenticate will time out at 1014.516843s 2024-03-29T10:50:03.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.218+0000 7f5b61508640 1 -- 172.21.15.64:0/3761429974 learned_addr learned my addr 172.21.15.64:0/3761429974 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:03.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488875759 0 0) 0x7f5b5c13fbd0 con 0x7f5b5c142690 2024-03-29T10:50:03.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:50:03.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b48003520 con 0x7f5b5c142690 2024-03-29T10:50:03.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445368372 0 0) 0x7f5b48003520 con 0x7f5b5c142690 2024-03-29T10:50:03.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 >> v1:172.21.15.174:6789/0 conn(0x7f5b5c1422b0 legacy=0x7f5b5c13f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:50:03.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.220+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 >> v1:172.21.15.64:6789/0 conn(0x7f5b5c0b17d0 legacy=0x7f5b5c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: _finish_hunting 0 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 1 monclient: found mon.noname-b 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c151940 con 0x7f5b5c142690 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: _finish_auth 0 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: _check_auth_tickets 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5b4c001690 con 0x7f5b5c142690 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.221+0000 7f5b60d07640 10 monclient: dump: 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b63f6d640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:50:03.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b63f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b4c0017f0 con 0x7f5b5c142690 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b60d07640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b60d07640 1 -- 172.21.15.64:0/3761429974 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5b4c001c40 con 0x7f5b5c142690 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b60d07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b60d07640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b60d07640 10 monclient: dump: 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.208 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b63f6d640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.222+0000 7f5b63f6d640 4 set_mon_vals no callback set 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.223+0000 7f5b63f6d640 10 monclient: shutdown 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.223+0000 7f5b63f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.223+0000 7f5b63f6d640 1 -- 172.21.15.64:0/3761429974 >> v1:172.21.15.64:6790/0 conn(0x7f5b5c142690 legacy=0x7f5b5c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.223+0000 7f5b63f6d640 1 -- 172.21.15.64:0/3761429974 shutdown_connections 2024-03-29T10:50:03.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.223+0000 7f5b63f6d640 1 -- 172.21.15.64:0/3761429974 >> 172.21.15.64:0/3761429974 conn(0x7f5b5c066a00 msgr2=0x7f5b5c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:03.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.226+0000 7f5b63f6d640 1 -- 172.21.15.64:0/3761429974 shutdown_connections 2024-03-29T10:50:03.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.227+0000 7f5b63f6d640 1 -- 172.21.15.64:0/3761429974 wait complete. 2024-03-29T10:50:03.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.228+0000 7f5b63f6d640 10 monclient: build_initial_monmap 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.228+0000 7f5b63f6d640 10 monclient: monmap: 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.228+0000 7f5b63f6d640 1 librados: starting msgr at 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.228+0000 7f5b63f6d640 1 librados: starting objecter 2024-03-29T10:50:03.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.228+0000 7f5b63f6d640 1 Processor -- start 2024-03-29T10:50:03.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.229+0000 7f5b63f6d640 1 -- start start 2024-03-29T10:50:03.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.229+0000 7f5b63f6d640 1 librados: setting wanted keys 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.229+0000 7f5b63f6d640 1 librados: calling monclient init 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.229+0000 7f5b63f6d640 10 monclient: init 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.229+0000 7f5b63f6d640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.229+0000 7f5b63f6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b5c0b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b5c1422b0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b5c142690 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0b6710 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c06e7d0 con 0x7f5b5c142690 2024-03-29T10:50:03.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c1e8610 con 0x7f5b5c1422b0 2024-03-29T10:50:03.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:03.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.231+0000 7f5b63f6d640 10 monclient(hunting): authenticate will time out at 1014.530499s 2024-03-29T10:50:03.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.233+0000 7f5b61d09640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5b5c0b17d0 0x7f5b5c06f420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45487/0 (socket says 172.21.15.64:45487) 2024-03-29T10:50:03.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.233+0000 7f5b61d09640 1 -- 172.21.15.64:0/229497045 learned_addr learned my addr 172.21.15.64:0/229497045 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:03.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.234+0000 7f5b527fc640 10 client.?.objecter ms_handle_connect 0x7f5b5c0b17d0 2024-03-29T10:50:03.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.234+0000 7f5b527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.234+0000 7f5b527fc640 10 client.?.objecter ms_handle_connect 0x7f5b5c142690 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.234+0000 7f5b527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618891595 0 0) 0x7f5b5c0b6710 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b40004eb0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805597312 0 0) 0x7f5b5c06e7d0 con 0x7f5b5c142690 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:50:03.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.235+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c0b6710 con 0x7f5b5c142690 2024-03-29T10:50:03.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.239+0000 7f5b527fc640 10 client.?.objecter ms_handle_connect 0x7f5b5c1422b0 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.239+0000 7f5b527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.239+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3790455596 0 0) 0x7f5b40004eb0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.239+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b5c06e7d0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.239+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044714824 0 0) 0x7f5b5c1e8610 con 0x7f5b5c1422b0 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.239+0000 7f5b527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b40004eb0 con 0x7f5b5c1422b0 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3771746684 0 0) 0x7f5b5c0b6710 con 0x7f5b5c142690 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b5c1e8610 con 0x7f5b5c142690 2024-03-29T10:50:03.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5b58001870 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): dump: 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.240+0000 7f5b527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:03.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.241+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5b54001ab0 con 0x7f5b5c142690 2024-03-29T10:50:03.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.241+0000 7f5b527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.241+0000 7f5b527fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.241+0000 7f5b527fc640 10 monclient(hunting): dump: 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.241+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3967704483 0 0) 0x7f5b5c06e7d0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 >> v1:172.21.15.64:6790/0 conn(0x7f5b5c1422b0 legacy=0x7f5b5c1e77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 >> v1:172.21.15.174:6789/0 conn(0x7f5b5c142690 legacy=0x7f5b5c1e7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 1 monclient: found mon.a 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c06ea40 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 5 monclient: authenticate success, global_id 4359 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 10 monclient: _renew_subs 2024-03-29T10:50:03.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c06ed20 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 10 monclient: _renew_subs 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c150ff0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.242+0000 7f5b63f6d640 1 librados: init done 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b580019d0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5b58004aa0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 10 monclient: dump: 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:03.232 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b53fff640 4 set_mon_vals no callback set 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.244+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f5b5801aa70 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f5b58084b00 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b527fc640 10 client.4359.objecter ms_dispatch 0x7f5b5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b527fc640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b527fc640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:03.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b527fc640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T10:50:03.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b61508640 10 monclient: get_auth_request con 0x7f5b4006ded0 auth_method 0 2024-03-29T10:50:03.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b47fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:03.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b47fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:03.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b47fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:03.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.245+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3c005080 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.248+0000 7f5b527fc640 10 client.4359.objecter ms_handle_connect 0x7f5b4006ded0 2024-03-29T10:50:03.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.255+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b58050480 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.255+0000 7f5b527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:03.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.255+0000 7f5b527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:03.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.584+0000 7f5b47fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:03.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.584+0000 7f5b47fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:03.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.584+0000 7f5b47fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:03.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.584+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b3c005330 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.585+0000 7f5b527fc640 1 -- 172.21.15.64:0/229497045 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f5b580523b0 con 0x7f5b5c0b17d0 2024-03-29T10:50:03.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.585+0000 7f5b527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:03.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.585+0000 7f5b527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:03.571 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:03.571 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:03.573 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.591+0000 7f5b47fff640 10 librados: watch_flush enter 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.591+0000 7f5b47fff640 10 librados: watch_flush exit 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 10 client.4359.objecter successfully canceled tick 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 >> v1:172.21.15.174:6816/2473791532 conn(0x7f5b4006ded0 legacy=0x7f5b40070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 10 monclient: shutdown 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:03.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 >> v1:172.21.15.64:6789/0 conn(0x7f5b5c0b17d0 legacy=0x7f5b5c06f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 shutdown_connections 2024-03-29T10:50:03.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.592+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 >> 172.21.15.64:0/229497045 conn(0x7f5b5c066a00 msgr2=0x7f5b5c13c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:03.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.593+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 shutdown_connections 2024-03-29T10:50:03.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.593+0000 7f5b47fff640 1 -- 172.21.15.64:0/229497045 wait complete. 2024-03-29T10:50:03.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:03.594+0000 7f5b47fff640 1 librados: shutdown 2024-03-29T10:50:03.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:10.401 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.566+0000 7f914bfff640 10 monclient: get_monmap_and_config 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.566+0000 7f914bfff640 10 monclient: build_initial_monmap 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.566+0000 7f914bfff640 10 monclient: monmap: 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:10.567005+0000 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:10.567005+0000 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:10.548 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:10.549 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:10.549 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:10.549 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:10.549 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:10.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.570+0000 7f914bfff640 1 Processor -- start 2024-03-29T10:50:10.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.570+0000 7f914bfff640 1 -- start start 2024-03-29T10:50:10.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.570+0000 7f914bfff640 10 monclient: init 2024-03-29T10:50:10.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:10.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:10.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f91440389e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:10.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f9144038dc0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f9144030660 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440c54c0 con 0x7f9144038dc0 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9144036430 con 0x7f9144030660 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9144039840 con 0x7f91440389e0 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.572+0000 7f914bfff640 10 monclient(hunting): authenticate will time out at 1021.871099s 2024-03-29T10:50:10.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.575+0000 7f914b7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f9144030660 0x7f91440d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:35363/0 (socket says 172.21.15.64:35363) 2024-03-29T10:50:10.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.575+0000 7f914b7fe640 1 -- 172.21.15.64:0/52700220 learned_addr learned my addr 172.21.15.64:0/52700220 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:10.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.577+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679914122 0 0) 0x7f9144036430 con 0x7f9144030660 2024-03-29T10:50:10.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.577+0000 7f9149ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:10.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.577+0000 7f9149ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:10.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.577+0000 7f9149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:10.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.577+0000 7f9149ffb640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:50:10.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.577+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9134003520 con 0x7f9144030660 2024-03-29T10:50:10.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.579+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1341343217 0 0) 0x7f9134003520 con 0x7f9144030660 2024-03-29T10:50:10.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.579+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 >> v1:172.21.15.64:6790/0 conn(0x7f91440389e0 legacy=0x7f9144033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:10.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 >> v1:172.21.15.64:6789/0 conn(0x7f9144038dc0 legacy=0x7f91440cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:10.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 1 monclient: found mon.noname-c 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144035690 con 0x7f9144030660 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f914c04f730 con 0x7f9144030660 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: dump: 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:10.562 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:10.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f914c075f10 con 0x7f9144030660 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 1 -- 172.21.15.64:0/52700220 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f914c0758b0 con 0x7f9144030660 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.580+0000 7f9149ffb640 10 monclient: dump: 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:10.564 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:10.565 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:10.565 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:10.565 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:10.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.583+0000 7f914bfff640 5 monclient: authenticate success, global_id 4387 2024-03-29T10:50:10.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.583+0000 7f914bfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:10.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.583+0000 7f914bfff640 4 set_mon_vals no callback set 2024-03-29T10:50:10.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.584+0000 7f914bfff640 10 monclient: shutdown 2024-03-29T10:50:10.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.584+0000 7f914bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:10.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.584+0000 7f914bfff640 1 -- 172.21.15.64:0/52700220 >> v1:172.21.15.174:6789/0 conn(0x7f9144030660 legacy=0x7f91440d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:10.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.584+0000 7f914bfff640 1 -- 172.21.15.64:0/52700220 shutdown_connections 2024-03-29T10:50:10.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.584+0000 7f914bfff640 1 -- 172.21.15.64:0/52700220 >> 172.21.15.64:0/52700220 conn(0x7f9144005c00 msgr2=0x7f9144006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:10.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.584+0000 7f914bfff640 1 -- 172.21.15.64:0/52700220 shutdown_connections 2024-03-29T10:50:10.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.585+0000 7f914bfff640 1 -- 172.21.15.64:0/52700220 wait complete. 2024-03-29T10:50:10.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.585+0000 7f914bfff640 10 monclient: build_initial_monmap 2024-03-29T10:50:10.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.585+0000 7f914bfff640 10 monclient: monmap: 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:10.568 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:10.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 1 librados: starting msgr at 2024-03-29T10:50:10.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 1 librados: starting objecter 2024-03-29T10:50:10.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 1 Processor -- start 2024-03-29T10:50:10.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 1 -- start start 2024-03-29T10:50:10.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 1 librados: setting wanted keys 2024-03-29T10:50:10.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 1 librados: calling monclient init 2024-03-29T10:50:10.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.586+0000 7f914bfff640 10 monclient: init 2024-03-29T10:50:10.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.587+0000 7f914bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:10.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.587+0000 7f914bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:10.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.587+0000 7f914bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f9144030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:10.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f91440389e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:10.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f91440c0c40 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:10.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440c8740 con 0x7f9144030660 2024-03-29T10:50:10.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914402eff0 con 0x7f91440389e0 2024-03-29T10:50:10.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440300e0 con 0x7f91440c0c40 2024-03-29T10:50:10.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:10.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.591+0000 7f914bfff640 10 monclient(hunting): authenticate will time out at 1021.890484s 2024-03-29T10:50:10.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.592+0000 7f914a7fc640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f91440389e0 0x7f914402f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:35447/0 (socket says 172.21.15.64:35447) 2024-03-29T10:50:10.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.592+0000 7f914a7fc640 1 -- 172.21.15.64:0/384866022 learned_addr learned my addr 172.21.15.64:0/384866022 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:10.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 10 client.?.objecter ms_handle_connect 0x7f91440389e0 2024-03-29T10:50:10.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80001700 0 0) 0x7f914402eff0 con 0x7f91440389e0 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.595+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9138004eb0 con 0x7f91440389e0 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.596+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3790072031 0 0) 0x7f9138004eb0 con 0x7f91440389e0 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.596+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f914402eff0 con 0x7f91440389e0 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f913c001930 con 0x7f91440389e0 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 10 monclient(hunting): dump: 2024-03-29T10:50:10.584 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1933650814 0 0) 0x7f914402eff0 con 0x7f91440389e0 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 >> v1:172.21.15.64:6790/0 conn(0x7f91440c0c40 legacy=0x7f91440bfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.597+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 >> v1:172.21.15.64:6789/0 conn(0x7f9144030660 legacy=0x7f914402d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:10.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 1 monclient: found mon.b 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914402d8d0 con 0x7f91440389e0 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f91337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 5 monclient: authenticate success, global_id 4390 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 10 monclient: _renew_subs 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91440c5960 con 0x7f91440389e0 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:10.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 10 monclient: _renew_subs 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91440c0600 con 0x7f91440389e0 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.598+0000 7f914bfff640 1 librados: init done 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.599+0000 7f91337fe640 10 client.4390.objecter ms_handle_connect 0x7f91440c0c40 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.599+0000 7f91337fe640 10 client.4390.objecter resend_mon_ops 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.599+0000 7f91337fe640 10 client.4390.objecter ms_handle_connect 0x7f9144030660 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.599+0000 7f91337fe640 10 client.4390.objecter resend_mon_ops 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.599+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f913c001d60 con 0x7f91440389e0 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f914bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f914bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f914bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9144007b90 con 0x7f91440389e0 2024-03-29T10:50:10.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f91337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f913c003a60 con 0x7f91440389e0 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f91337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f91337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f91337fe640 10 monclient: dump: 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:10.588 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:10.589 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.601+0000 7f91497fa640 4 set_mon_vals no callback set 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.606+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f913c003bc0 con 0x7f91440389e0 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.607+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f913c084a20 con 0x7f91440389e0 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.607+0000 7f91337fe640 10 client.4390.objecter ms_dispatch 0x7f914c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.607+0000 7f914affd640 10 monclient: get_auth_request con 0x7f913806dc10 auth_method 0 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.607+0000 7f91337fe640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.607+0000 7f91337fe640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.607+0000 7f91337fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T10:50:10.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.608+0000 7f91337fe640 10 client.4390.objecter ms_handle_connect 0x7f913806dc10 2024-03-29T10:50:10.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.612+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f913c050300 con 0x7f91440389e0 2024-03-29T10:50:10.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.612+0000 7f91337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:10.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.612+0000 7f91337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:10.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.945+0000 7f914bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:10.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.945+0000 7f914bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:10.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.945+0000 7f914bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:10.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.945+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f914402f480 con 0x7f91440389e0 2024-03-29T10:50:10.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.946+0000 7f91337fe640 1 -- 172.21.15.64:0/384866022 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f914402f480 con 0x7f91440389e0 2024-03-29T10:50:10.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.946+0000 7f91337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:10.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.946+0000 7f91337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:10.928 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:10.928 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:10.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.950+0000 7f914bfff640 10 librados: watch_flush enter 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.951+0000 7f914bfff640 10 librados: watch_flush exit 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.951+0000 7f914bfff640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.951+0000 7f914bfff640 10 client.4390.objecter successfully canceled tick 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.952+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 >> v1:172.21.15.174:6816/2473791532 conn(0x7f913806dc10 legacy=0x7f91380700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.952+0000 7f914bfff640 10 monclient: shutdown 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.952+0000 7f914bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:10.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.952+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 >> v1:172.21.15.174:6789/0 conn(0x7f91440389e0 legacy=0x7f914402f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:10.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.954+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 shutdown_connections 2024-03-29T10:50:10.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.954+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 >> 172.21.15.64:0/384866022 conn(0x7f9144005c00 msgr2=0x7f9144034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:10.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.954+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 shutdown_connections 2024-03-29T10:50:10.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.955+0000 7f914bfff640 1 -- 172.21.15.64:0/384866022 wait complete. 2024-03-29T10:50:10.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:10.955+0000 7f914bfff640 1 librados: shutdown 2024-03-29T10:50:10.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:17.764 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:17.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.934+0000 7fef539af640 10 monclient: get_monmap_and_config 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.934+0000 7fef539af640 10 monclient: build_initial_monmap 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.934+0000 7fef539af640 10 monclient: monmap: 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:17.935727+0000 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:17.935727+0000 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:17.918 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.935+0000 7fef539af640 1 Processor -- start 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.936+0000 7fef539af640 1 -- start start 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.936+0000 7fef539af640 10 monclient: init 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.937+0000 7fef539af640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.937+0000 7fef539af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:17.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.937+0000 7fef539af640 10 monclient(hunting): picked mon.noname-b con 0x7fef4c0b4100 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:17.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 10 monclient(hunting): picked mon.noname-a con 0x7fef4c13ee90 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:17.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 10 monclient(hunting): picked mon.noname-c con 0x7fef4c0af900 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:17.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c0afd10 con 0x7fef4c13ee90 2024-03-29T10:50:17.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c0af6f0 con 0x7fef4c0af900 2024-03-29T10:50:17.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c0aaf60 con 0x7fef4c0b4100 2024-03-29T10:50:17.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:17.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.939+0000 7fef539af640 10 monclient(hunting): authenticate will time out at 1029.237986s 2024-03-29T10:50:17.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.941+0000 7fef50f4a640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fef4c13ee90 0x7fef4c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33269/0 (socket says 172.21.15.64:33269) 2024-03-29T10:50:17.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.941+0000 7fef50f4a640 1 -- 172.21.15.64:0/1459459737 learned_addr learned my addr 172.21.15.64:0/1459459737 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:17.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122207694 0 0) 0x7fef4c0afd10 con 0x7fef4c13ee90 2024-03-29T10:50:17.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:17.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:17.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:17.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38003520 con 0x7fef4c13ee90 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254174775 0 0) 0x7fef38003520 con 0x7fef4c13ee90 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 >> v1:172.21.15.64:6790/0 conn(0x7fef4c0b4100 legacy=0x7fef4c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 >> v1:172.21.15.174:6789/0 conn(0x7fef4c0af900 legacy=0x7fef4c13b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 monclient: found mon.noname-a 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c13e970 con 0x7fef4c13ee90 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: _finish_auth 0 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fef44001a10 con 0x7fef4c13ee90 2024-03-29T10:50:17.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.942+0000 7fef43fff640 10 monclient: dump: 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.943+0000 7fef539af640 5 monclient: authenticate success, global_id 4368 2024-03-29T10:50:17.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.943+0000 7fef539af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.943+0000 7fef43fff640 1 -- 172.21.15.64:0/1459459737 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef44001be0 con 0x7fef4c13ee90 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.943+0000 7fef43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.943+0000 7fef539af640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.943+0000 7fef539af640 4 set_mon_vals no callback set 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.944+0000 7fef539af640 10 monclient: shutdown 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.944+0000 7fef539af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.944+0000 7fef539af640 1 -- 172.21.15.64:0/1459459737 >> v1:172.21.15.64:6789/0 conn(0x7fef4c13ee90 legacy=0x7fef4c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.944+0000 7fef539af640 1 -- 172.21.15.64:0/1459459737 shutdown_connections 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.944+0000 7fef539af640 1 -- 172.21.15.64:0/1459459737 >> 172.21.15.64:0/1459459737 conn(0x7fef4c0b35e0 msgr2=0x7fef4c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:17.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.944+0000 7fef539af640 1 -- 172.21.15.64:0/1459459737 shutdown_connections 2024-03-29T10:50:17.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.945+0000 7fef539af640 1 -- 172.21.15.64:0/1459459737 wait complete. 2024-03-29T10:50:17.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.945+0000 7fef539af640 10 monclient: build_initial_monmap 2024-03-29T10:50:17.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.945+0000 7fef539af640 10 monclient: monmap: 2024-03-29T10:50:17.930 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:17.930 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 1 librados: starting msgr at 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 1 librados: starting objecter 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 1 Processor -- start 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 1 -- start start 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 1 librados: setting wanted keys 2024-03-29T10:50:17.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 1 librados: calling monclient init 2024-03-29T10:50:17.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.946+0000 7fef539af640 10 monclient: init 2024-03-29T10:50:17.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:17.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:17.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient(hunting): picked mon.noname-c con 0x7fef4c0af900 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:17.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient(hunting): picked mon.noname-b con 0x7fef4c0b4100 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:17.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient(hunting): picked mon.noname-a con 0x7fef4c143310 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:17.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c072690 con 0x7fef4c143310 2024-03-29T10:50:17.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c071a60 con 0x7fef4c0b4100 2024-03-29T10:50:17.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4c140700 con 0x7fef4c0af900 2024-03-29T10:50:17.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:17.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.947+0000 7fef539af640 10 monclient(hunting): authenticate will time out at 1029.246484s 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.955+0000 7fef50f4a640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fef4c0b4100 0x7fef4c132880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:40367/0 (socket says 172.21.15.64:40367) 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.955+0000 7fef50f4a640 1 -- 172.21.15.64:0/3210283868 learned_addr learned my addr 172.21.15.64:0/3210283868 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 client.?.objecter ms_handle_connect 0x7fef4c0b4100 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 client.?.objecter ms_handle_connect 0x7fef4c0af900 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 client.?.objecter ms_handle_connect 0x7fef4c143310 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516224552 0 0) 0x7fef4c071a60 con 0x7fef4c0b4100 2024-03-29T10:50:17.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:17.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:17.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:17.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:50:17.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.956+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef30004eb0 con 0x7fef4c0b4100 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648366027 0 0) 0x7fef4c072690 con 0x7fef4c143310 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef4c071a60 con 0x7fef4c143310 2024-03-29T10:50:17.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4108881951 0 0) 0x7fef4c071a60 con 0x7fef4c143310 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef4c072690 con 0x7fef4c143310 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fef3c001790 con 0x7fef4c143310 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:17.942 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.957+0000 7fef41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3423768890 0 0) 0x7fef4c072690 con 0x7fef4c143310 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 >> v1:172.21.15.64:6790/0 conn(0x7fef4c0af900 legacy=0x7fef4c071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 >> v1:172.21.15.174:6789/0 conn(0x7fef4c0b4100 legacy=0x7fef4c132880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 monclient: found mon.a 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c132f90 con 0x7fef4c143310 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:17.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef3c001c80 con 0x7fef4c143310 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fef3c003830 con 0x7fef4c143310 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.958+0000 7fef42ffd640 4 set_mon_vals no callback set 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.959+0000 7fef41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.959+0000 7fef41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.959+0000 7fef41ffb640 10 monclient: dump: 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:17.944 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:17.945 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:17.945 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 5 monclient: authenticate success, global_id 4371 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 10 monclient: _renew_subs 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c1df020 con 0x7fef4c143310 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 10 monclient: _renew_subs 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4c1df5a0 con 0x7fef4c143310 2024-03-29T10:50:17.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.960+0000 7fef539af640 1 librados: init done 2024-03-29T10:50:17.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.963+0000 7fef539af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:17.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.963+0000 7fef539af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:17.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.963+0000 7fef539af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:17.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.963+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4c06dfd0 con 0x7fef4c143310 2024-03-29T10:50:17.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.966+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fef3c019800 con 0x7fef4c143310 2024-03-29T10:50:17.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.966+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fef3c083730 con 0x7fef4c143310 2024-03-29T10:50:17.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.966+0000 7fef41ffb640 10 client.4371.objecter ms_dispatch 0x7fef4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:17.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.966+0000 7fef41ffb640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:17.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.966+0000 7fef41ffb640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:17.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.966+0000 7fef41ffb640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-29T10:50:17.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.967+0000 7fef5174b640 10 monclient: get_auth_request con 0x7fef3006dde0 auth_method 0 2024-03-29T10:50:17.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.968+0000 7fef41ffb640 10 client.4371.objecter ms_handle_connect 0x7fef3006dde0 2024-03-29T10:50:17.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.974+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef3c04f010 con 0x7fef4c143310 2024-03-29T10:50:17.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.975+0000 7fef41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:17.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:17.975+0000 7fef41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:18.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef539af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef539af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef539af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef4c1df1b0 con 0x7fef4c143310 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef41ffb640 1 -- 172.21.15.64:0/3210283868 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fef3c050f40 con 0x7fef4c143310 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.321+0000 7fef41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:18.304 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:18.305 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:18.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 10 librados: watch_flush enter 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 10 librados: watch_flush exit 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 10 client.4371.objecter successfully canceled tick 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 >> v1:172.21.15.174:6816/2473791532 conn(0x7fef3006dde0 legacy=0x7fef300702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 10 monclient: shutdown 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 >> v1:172.21.15.64:6789/0 conn(0x7fef4c143310 legacy=0x7fef4c1de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 shutdown_connections 2024-03-29T10:50:18.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 >> 172.21.15.64:0/3210283868 conn(0x7fef4c0b35e0 msgr2=0x7fef4c0ac920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:18.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 shutdown_connections 2024-03-29T10:50:18.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.325+0000 7fef539af640 1 -- 172.21.15.64:0/3210283868 wait complete. 2024-03-29T10:50:18.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:18.326+0000 7fef539af640 1 librados: shutdown 2024-03-29T10:50:18.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:25.134 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:25.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.300+0000 7fe038074640 10 monclient: get_monmap_and_config 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.300+0000 7fe038074640 10 monclient: build_initial_monmap 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.300+0000 7fe038074640 10 monclient: monmap: 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:25.301916+0000 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:25.301916+0000 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:25.283 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.300+0000 7fe038074640 1 Processor -- start 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.301+0000 7fe038074640 1 -- start start 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.301+0000 7fe038074640 10 monclient: init 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient(hunting): picked mon.noname-b con 0x7fe0300a9750 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:25.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient(hunting): picked mon.noname-a con 0x7fe03013e1e0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:25.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient(hunting): picked mon.noname-c con 0x7fe030071c30 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:25.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0300ab300 con 0x7fe03013e1e0 2024-03-29T10:50:25.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0300ac2f0 con 0x7fe030071c30 2024-03-29T10:50:25.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0300ad8d0 con 0x7fe0300a9750 2024-03-29T10:50:25.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:25.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.302+0000 7fe038074640 10 monclient(hunting): authenticate will time out at 1036.602063s 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.303+0000 7fe035e10640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe0300a9750 0x7fe03013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33442/0 (socket says 172.21.15.64:33442) 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.303+0000 7fe035e10640 1 -- 172.21.15.64:0/2435054889 learned_addr learned my addr 172.21.15.64:0/2435054889 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.304+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998099399 0 0) 0x7fe0300ad8d0 con 0x7fe0300a9750 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.304+0000 7fe034e0e640 10 monclient(hunting): _negotiate 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.304+0000 7fe034e0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.304+0000 7fe034e0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:25.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.304+0000 7fe034e0e640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.304+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003520 con 0x7fe0300a9750 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.305+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571667491 0 0) 0x7fe01c003520 con 0x7fe0300a9750 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.305+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 >> v1:172.21.15.174:6789/0 conn(0x7fe030071c30 legacy=0x7fe0300ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.305+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 >> v1:172.21.15.64:6789/0 conn(0x7fe03013e1e0 legacy=0x7fe0300ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: _finish_hunting 0 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 1 monclient: found mon.noname-b 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0300ada60 con 0x7fe0300a9750 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: _finish_auth 0 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: _check_auth_tickets 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe038074640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:50:25.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe038074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe02c001690 con 0x7fe0300a9750 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe034e0e640 10 monclient: dump: 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:25.296 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.307+0000 7fe038074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.308+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe02c001920 con 0x7fe0300a9750 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.308+0000 7fe034e0e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.308+0000 7fe038074640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.308+0000 7fe038074640 4 set_mon_vals no callback set 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe034e0e640 1 -- 172.21.15.64:0/2435054889 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe02c001cb0 con 0x7fe0300a9750 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe034e0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe034e0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe034e0e640 10 monclient: dump: 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.297 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe038074640 10 monclient: shutdown 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe038074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:25.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.309+0000 7fe038074640 1 -- 172.21.15.64:0/2435054889 >> v1:172.21.15.64:6790/0 conn(0x7fe0300a9750 legacy=0x7fe03013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:25.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.312+0000 7fe038074640 1 -- 172.21.15.64:0/2435054889 shutdown_connections 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.312+0000 7fe038074640 1 -- 172.21.15.64:0/2435054889 >> 172.21.15.64:0/2435054889 conn(0x7fe03006b660 msgr2=0x7fe03006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.312+0000 7fe038074640 1 -- 172.21.15.64:0/2435054889 shutdown_connections 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.312+0000 7fe038074640 1 -- 172.21.15.64:0/2435054889 wait complete. 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.313+0000 7fe038074640 10 monclient: build_initial_monmap 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.313+0000 7fe038074640 10 monclient: monmap: 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:25.301 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.314+0000 7fe038074640 1 librados: starting msgr at 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.314+0000 7fe038074640 1 librados: starting objecter 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.314+0000 7fe038074640 1 Processor -- start 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.316+0000 7fe038074640 1 -- start start 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.316+0000 7fe038074640 1 librados: setting wanted keys 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.316+0000 7fe038074640 1 librados: calling monclient init 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.316+0000 7fe038074640 10 monclient: init 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient(hunting): picked mon.noname-c con 0x7fe030071c30 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient(hunting): picked mon.noname-b con 0x7fe0300a9750 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:25.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient(hunting): picked mon.noname-a con 0x7fe03013e1e0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:25.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe030070b70 con 0x7fe03013e1e0 2024-03-29T10:50:25.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0301e4640 con 0x7fe0300a9750 2024-03-29T10:50:25.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0301e4160 con 0x7fe030071c30 2024-03-29T10:50:25.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:25.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.317+0000 7fe038074640 10 monclient(hunting): authenticate will time out at 1036.617422s 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.318+0000 7fe035e10640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe030071c30 0x7fe03006e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35644/0 (socket says 172.21.15.64:35644) 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.318+0000 7fe035e10640 1 -- 172.21.15.64:0/1403976134 learned_addr learned my addr 172.21.15.64:0/1403976134 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 10 client.?.objecter ms_handle_connect 0x7fe030071c30 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699202287 0 0) 0x7fe0301e4160 con 0x7fe030071c30 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:25.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.321+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe014004eb0 con 0x7fe030071c30 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 10 client.?.objecter ms_handle_connect 0x7fe03013e1e0 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386935355 0 0) 0x7fe030070b70 con 0x7fe03013e1e0 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:50:25.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.322+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0301e4160 con 0x7fe03013e1e0 2024-03-29T10:50:25.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.323+0000 7fe0267fc640 10 client.?.objecter ms_handle_connect 0x7fe0300a9750 2024-03-29T10:50:25.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.323+0000 7fe0267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:25.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.323+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177868696 0 0) 0x7fe0301e4640 con 0x7fe0300a9750 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe030070b70 con 0x7fe0300a9750 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024147687 0 0) 0x7fe0301e4160 con 0x7fe03013e1e0 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0301e4640 con 0x7fe03013e1e0 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe028001b30 con 0x7fe03013e1e0 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): dump: 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:25.309 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.324+0000 7fe0267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4229673447 0 0) 0x7fe0301e4640 con 0x7fe03013e1e0 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 >> v1:172.21.15.64:6790/0 conn(0x7fe030071c30 legacy=0x7fe03006e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 >> v1:172.21.15.174:6789/0 conn(0x7fe0300a9750 legacy=0x7fe03006ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:25.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 1 monclient: found mon.a 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0301e42f0 con 0x7fe03013e1e0 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe0267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe038074640 5 monclient: authenticate success, global_id 4374 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe038074640 10 monclient: _renew_subs 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.325+0000 7fe038074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.326+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03013a040 con 0x7fe03013e1e0 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.326+0000 7fe038074640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.326+0000 7fe038074640 10 monclient: _renew_subs 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.326+0000 7fe038074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:25.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.326+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0300714e0 con 0x7fe03013e1e0 2024-03-29T10:50:25.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.326+0000 7fe038074640 1 librados: init done 2024-03-29T10:50:25.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe028002040 con 0x7fe03013e1e0 2024-03-29T10:50:25.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:25.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe028003bb0 con 0x7fe03013e1e0 2024-03-29T10:50:25.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:25.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:25.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 10 monclient: dump: 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe028019b80 con 0x7fe03013e1e0 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe038074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe038074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:25.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe038074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03013a3e0 con 0x7fe03013e1e0 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.329+0000 7fe027fff640 4 set_mon_vals no callback set 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.330+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe028083ba0 con 0x7fe03013e1e0 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.330+0000 7fe0267fc640 10 client.4374.objecter ms_dispatch 0x7fe030000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.330+0000 7fe0267fc640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.330+0000 7fe0267fc640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.330+0000 7fe0267fc640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T10:50:25.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.330+0000 7fe035e10640 10 monclient: get_auth_request con 0x7fe01406dc90 auth_method 0 2024-03-29T10:50:25.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.332+0000 7fe0267fc640 10 client.4374.objecter ms_handle_connect 0x7fe01406dc90 2024-03-29T10:50:25.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.337+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe02804f330 con 0x7fe03013e1e0 2024-03-29T10:50:25.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.337+0000 7fe0267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:25.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.337+0000 7fe0267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:25.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.665+0000 7fe038074640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:25.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.665+0000 7fe038074640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:25.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.665+0000 7fe038074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:25.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.665+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0301e47d0 con 0x7fe03013e1e0 2024-03-29T10:50:25.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.667+0000 7fe0267fc640 1 -- 172.21.15.64:0/1403976134 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe028051430 con 0x7fe03013e1e0 2024-03-29T10:50:25.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.667+0000 7fe0267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:25.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.667+0000 7fe0267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:25.651 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:25.652 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:25.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.671+0000 7fe038074640 10 librados: watch_flush enter 2024-03-29T10:50:25.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.671+0000 7fe038074640 10 librados: watch_flush exit 2024-03-29T10:50:25.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.671+0000 7fe038074640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T10:50:25.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.671+0000 7fe038074640 10 client.4374.objecter successfully canceled tick 2024-03-29T10:50:25.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe01406dc90 legacy=0x7fe014070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:25.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 10 monclient: shutdown 2024-03-29T10:50:25.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:25.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 >> v1:172.21.15.64:6789/0 conn(0x7fe03013e1e0 legacy=0x7fe03006f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:25.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 shutdown_connections 2024-03-29T10:50:25.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 >> 172.21.15.64:0/1403976134 conn(0x7fe03006b660 msgr2=0x7fe03013b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:25.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 shutdown_connections 2024-03-29T10:50:25.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 -- 172.21.15.64:0/1403976134 wait complete. 2024-03-29T10:50:25.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:25.672+0000 7fe038074640 1 librados: shutdown 2024-03-29T10:50:25.672 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:32.479 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.673+0000 7fe9c19cd640 10 monclient: get_monmap_and_config 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.673+0000 7fe9c19cd640 10 monclient: build_initial_monmap 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.673+0000 7fe9c19cd640 10 monclient: monmap: 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:32.674569+0000 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:32.674569+0000 2024-03-29T10:50:32.656 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.674+0000 7fe9c19cd640 1 Processor -- start 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.674+0000 7fe9c19cd640 1 -- start start 2024-03-29T10:50:32.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.674+0000 7fe9c19cd640 10 monclient: init 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient(hunting): picked mon.noname-a con 0x7fe9bc06ade0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient(hunting): picked mon.noname-c con 0x7fe9bc0afa30 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient(hunting): picked mon.noname-b con 0x7fe9bc1433e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc072670 con 0x7fe9bc06ade0 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc13da60 con 0x7fe9bc0afa30 2024-03-29T10:50:32.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc143fa0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9ba7fc640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe9bc0afa30 0x7fe9bc13d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:43908/0 (socket says 172.21.15.64:43908) 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9ba7fc640 1 -- 172.21.15.64:0/174429151 learned_addr learned my addr 172.21.15.64:0/174429151 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.675+0000 7fe9c19cd640 10 monclient(hunting): authenticate will time out at 1043.974734s 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917880115 0 0) 0x7fe9bc13da60 con 0x7fe9bc0afa30 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a4003520 con 0x7fe9bc0afa30 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169391969 0 0) 0x7fe9bc143fa0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc13da60 con 0x7fe9bc1433e0 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479229862 0 0) 0x7fe9a4003520 con 0x7fe9bc0afa30 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.676+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 >> v1:172.21.15.64:6790/0 conn(0x7fe9bc1433e0 legacy=0x7fe9bc143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 >> v1:172.21.15.64:6789/0 conn(0x7fe9bc06ade0 legacy=0x7fe9bc071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 1 monclient: found mon.noname-c 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc14f730 con 0x7fe9bc0afa30 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:32.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9ac001690 con 0x7fe9bc0afa30 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: dump: 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:32.664 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9c19cd640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9c19cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9ac0018b0 con 0x7fe9bc0afa30 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 1 -- 172.21.15.64:0/174429151 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9ac001bc0 con 0x7fe9bc0afa30 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9b9ffb640 10 monclient: dump: 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.665 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9c19cd640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.677+0000 7fe9c19cd640 4 set_mon_vals no callback set 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.678+0000 7fe9c19cd640 10 monclient: shutdown 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.678+0000 7fe9c19cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.678+0000 7fe9c19cd640 1 -- 172.21.15.64:0/174429151 >> v1:172.21.15.174:6789/0 conn(0x7fe9bc0afa30 legacy=0x7fe9bc13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.678+0000 7fe9c19cd640 1 -- 172.21.15.64:0/174429151 shutdown_connections 2024-03-29T10:50:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.678+0000 7fe9c19cd640 1 -- 172.21.15.64:0/174429151 >> 172.21.15.64:0/174429151 conn(0x7fe9bc067470 msgr2=0x7fe9bc0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.678+0000 7fe9c19cd640 1 -- 172.21.15.64:0/174429151 shutdown_connections 2024-03-29T10:50:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.679+0000 7fe9c19cd640 1 -- 172.21.15.64:0/174429151 wait complete. 2024-03-29T10:50:32.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.680+0000 7fe9c19cd640 10 monclient: build_initial_monmap 2024-03-29T10:50:32.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.680+0000 7fe9c19cd640 10 monclient: monmap: 2024-03-29T10:50:32.672 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:32.672 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:32.672 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.680+0000 7fe9c19cd640 1 librados: starting msgr at 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.680+0000 7fe9c19cd640 1 librados: starting objecter 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.681+0000 7fe9c19cd640 1 Processor -- start 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.681+0000 7fe9c19cd640 1 -- start start 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.681+0000 7fe9c19cd640 1 librados: setting wanted keys 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.681+0000 7fe9c19cd640 1 librados: calling monclient init 2024-03-29T10:50:32.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.681+0000 7fe9c19cd640 10 monclient: init 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.682+0000 7fe9c19cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.682+0000 7fe9c19cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.682+0000 7fe9c19cd640 10 monclient(hunting): picked mon.noname-a con 0x7fe9bc06ade0 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.682+0000 7fe9c19cd640 10 monclient(hunting): picked mon.noname-c con 0x7fe9bc0afa30 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.682+0000 7fe9ba7fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe9bc0afa30 0x7fe9bc06fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40048/0 (socket says 172.21.15.64:40048) 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.682+0000 7fe9ba7fc640 1 -- 172.21.15.64:0/136439287 learned_addr learned my addr 172.21.15.64:0/136439287 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9a3fff640 10 client.?.objecter ms_handle_connect 0x7fe9bc0afa30 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9c19cd640 10 monclient(hunting): picked mon.noname-b con 0x7fe9bc1433e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc139470 con 0x7fe9bc06ade0 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9a3fff640 10 client.?.objecter ms_handle_connect 0x7fe9bc06ade0 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:32.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9a3fff640 10 client.?.objecter ms_handle_connect 0x7fe9bc1433e0 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc139900 con 0x7fe9bc1433e0 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc13a490 con 0x7fe9bc0afa30 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9c19cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.683+0000 7fe9c19cd640 10 monclient(hunting): authenticate will time out at 1043.982659s 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155354549 0 0) 0x7fe9bc139900 con 0x7fe9bc1433e0 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a8004eb0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228820538 0 0) 0x7fe9bc13a490 con 0x7fe9bc0afa30 2024-03-29T10:50:32.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc139900 con 0x7fe9bc0afa30 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276135710 0 0) 0x7fe9bc139470 con 0x7fe9bc06ade0 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.684+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc13a490 con 0x7fe9bc06ade0 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 662528790 0 0) 0x7fe9bc13a490 con 0x7fe9bc06ade0 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a8005620 con 0x7fe9bc06ade0 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168697442 0 0) 0x7fe9a8004eb0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a8005be0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9b4001bd0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): dump: 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.677 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9b00017e0 con 0x7fe9bc06ade0 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): dump: 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:32.678 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963024799 0 0) 0x7fe9bc139900 con 0x7fe9bc0afa30 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a8004eb0 con 0x7fe9bc0afa30 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9ac001590 con 0x7fe9bc0afa30 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:32.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient(hunting): dump: 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 348068787 0 0) 0x7fe9a8005be0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 >> v1:172.21.15.64:6790/0 conn(0x7fe9bc0afa30 legacy=0x7fe9bc06fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 >> v1:172.21.15.64:6789/0 conn(0x7fe9bc06ade0 legacy=0x7fe9bc138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 monclient: found mon.b 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc13a770 con 0x7fe9bc1433e0 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9c19cd640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9c19cd640 10 monclient: _renew_subs 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9c19cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.685+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc138ec0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9c19cd640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9c19cd640 10 monclient: _renew_subs 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9c19cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc0704b0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9c19cd640 1 librados: init done 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b40020c0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe9b4003b10 con 0x7fe9bc1433e0 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9a3fff640 10 monclient: dump: 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.682 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.686+0000 7fe9b97fa640 4 set_mon_vals no callback set 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.687+0000 7fe9c19cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.688+0000 7fe9c19cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.688+0000 7fe9c19cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.688+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc075f10 con 0x7fe9bc1433e0 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.688+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe9b4019ae0 con 0x7fe9bc1433e0 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.689+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe9b4083b00 con 0x7fe9bc1433e0 2024-03-29T10:50:32.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.689+0000 7fe9a3fff640 10 client.4402.objecter ms_dispatch 0x7fe9bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:32.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.689+0000 7fe9a3fff640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:32.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.689+0000 7fe9a3fff640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:32.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.689+0000 7fe9a3fff640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:50:32.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.689+0000 7fe9baffd640 10 monclient: get_auth_request con 0x7fe9a806e630 auth_method 0 2024-03-29T10:50:32.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.690+0000 7fe9a3fff640 10 client.4402.objecter ms_handle_connect 0x7fe9a806e630 2024-03-29T10:50:32.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.695+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9b404f210 con 0x7fe9bc1433e0 2024-03-29T10:50:32.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.695+0000 7fe9a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:32.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:32.695+0000 7fe9a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:33.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.066+0000 7fe9c19cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:33.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.066+0000 7fe9c19cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:33.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.066+0000 7fe9c19cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:33.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.066+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9bc139620 con 0x7fe9bc1433e0 2024-03-29T10:50:33.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.067+0000 7fe9a3fff640 1 -- 172.21.15.64:0/136439287 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe9bc139620 con 0x7fe9bc1433e0 2024-03-29T10:50:33.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.067+0000 7fe9a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:33.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.067+0000 7fe9a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:33.049 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:33.049 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.070+0000 7fe9c19cd640 10 librados: watch_flush enter 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 10 librados: watch_flush exit 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe9a806e630 legacy=0x7fe9a8070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 10 monclient: shutdown 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 >> v1:172.21.15.174:6789/0 conn(0x7fe9bc1433e0 legacy=0x7fe9bc136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 shutdown_connections 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 >> 172.21.15.64:0/136439287 conn(0x7fe9bc067470 msgr2=0x7fe9bc0b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:33.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.071+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 shutdown_connections 2024-03-29T10:50:33.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.072+0000 7fe9c19cd640 1 -- 172.21.15.64:0/136439287 wait complete. 2024-03-29T10:50:33.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:33.072+0000 7fe9c19cd640 1 librados: shutdown 2024-03-29T10:50:33.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:39.874 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:40.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.037+0000 7fc8c2756640 10 monclient: get_monmap_and_config 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.037+0000 7fc8c2756640 10 monclient: build_initial_monmap 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.037+0000 7fc8c2756640 10 monclient: monmap: 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:40.038708+0000 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:40.038708+0000 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:40.020 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.038+0000 7fc8c2756640 1 Processor -- start 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.039+0000 7fc8c2756640 1 -- start start 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.039+0000 7fc8c2756640 10 monclient: init 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.039+0000 7fc8c2756640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.039+0000 7fc8c2756640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.039+0000 7fc8c2756640 10 monclient(hunting): picked mon.noname-b con 0x7fc8bc1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.039+0000 7fc8c2756640 10 monclient(hunting): picked mon.noname-a con 0x7fc8bc13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8c2756640 10 monclient(hunting): picked mon.noname-c con 0x7fc8bc0a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8c2756640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0aaef0 con 0x7fc8bc13f650 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8c2756640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc145bd0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8c2756640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc145eb0 con 0x7fc8bc1457f0 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8c2756640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8c2756640 10 monclient(hunting): authenticate will time out at 1051.339067s 2024-03-29T10:50:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8bbfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc8bc1457f0 0x7fc8bc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52662/0 (socket says 172.21.15.64:52662) 2024-03-29T10:50:40.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.040+0000 7fc8bbfff640 1 -- 172.21.15.64:0/3523508831 learned_addr learned my addr 172.21.15.64:0/3523508831 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:40.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915016969 0 0) 0x7fc8bc145eb0 con 0x7fc8bc1457f0 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0003520 con 0x7fc8bc1457f0 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539350478 0 0) 0x7fc8bc0aaef0 con 0x7fc8bc13f650 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:40.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc145eb0 con 0x7fc8bc13f650 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 940231002 0 0) 0x7fc8bc145eb0 con 0x7fc8bc13f650 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 >> v1:172.21.15.64:6790/0 conn(0x7fc8bc1457f0 legacy=0x7fc8bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 >> v1:172.21.15.174:6789/0 conn(0x7fc8bc0a9d70 legacy=0x7fc8bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 1 monclient: found mon.noname-a 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.041+0000 7fc8baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc146340 con 0x7fc8bc13f650 2024-03-29T10:50:40.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc8a8001670 con 0x7fc8bc13f650 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: dump: 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:40.048 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8a80018c0 con 0x7fc8bc13f650 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 1 -- 172.21.15.64:0/3523508831 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc8a8001c50 con 0x7fc8bc13f650 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8baffd640 10 monclient: dump: 2024-03-29T10:50:40.049 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8c2756640 5 monclient: authenticate success, global_id 4383 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.042+0000 7fc8c2756640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:40.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.043+0000 7fc8c2756640 4 set_mon_vals no callback set 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.043+0000 7fc8c2756640 10 monclient: shutdown 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.043+0000 7fc8c2756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.043+0000 7fc8c2756640 1 -- 172.21.15.64:0/3523508831 >> v1:172.21.15.64:6789/0 conn(0x7fc8bc13f650 legacy=0x7fc8bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.044+0000 7fc8c2756640 1 -- 172.21.15.64:0/3523508831 shutdown_connections 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.044+0000 7fc8c2756640 1 -- 172.21.15.64:0/3523508831 >> 172.21.15.64:0/3523508831 conn(0x7fc8bc066a00 msgr2=0x7fc8bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.044+0000 7fc8c2756640 1 -- 172.21.15.64:0/3523508831 shutdown_connections 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.044+0000 7fc8c2756640 1 -- 172.21.15.64:0/3523508831 wait complete. 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.045+0000 7fc8c2756640 10 monclient: build_initial_monmap 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.045+0000 7fc8c2756640 10 monclient: monmap: 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:40.051 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:40.052 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:40.052 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:40.052 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:40.052 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:40.052 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.045+0000 7fc8c2756640 1 librados: starting msgr at 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.045+0000 7fc8c2756640 1 librados: starting objecter 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.045+0000 7fc8c2756640 1 Processor -- start 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 1 -- start start 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 1 librados: setting wanted keys 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 1 librados: calling monclient init 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient: init 2024-03-29T10:50:40.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient(hunting): picked mon.noname-b con 0x7fc8bc0a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient(hunting): picked mon.noname-a con 0x7fc8bc13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient(hunting): picked mon.noname-c con 0x7fc8bc1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc139a20 con 0x7fc8bc13f650 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc144000 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc144c30 con 0x7fc8bc1457f0 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.046+0000 7fc8c2756640 10 monclient(hunting): authenticate will time out at 1051.345817s 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8c0cf3640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc8bc1457f0 0x7fc8bc144520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53332/0 (socket says 172.21.15.64:53332) 2024-03-29T10:50:40.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8c0cf3640 1 -- 172.21.15.64:0/3038935738 learned_addr learned my addr 172.21.15.64:0/3038935738 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fc8bc1457f0 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fc8bc0a9d70 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294287316 0 0) 0x7fc8bc144000 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a4004eb0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fc8bc13f650 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.047+0000 7fc8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33130510 0 0) 0x7fc8bc144c30 con 0x7fc8bc1457f0 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc144000 con 0x7fc8bc1457f0 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4139454855 0 0) 0x7fc8a4004eb0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8bc144c30 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418662772 0 0) 0x7fc8bc139a20 con 0x7fc8bc13f650 2024-03-29T10:50:40.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a4004eb0 con 0x7fc8bc13f650 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc8b4001af0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:40.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604141990 0 0) 0x7fc8bc144c30 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 >> v1:172.21.15.64:6790/0 conn(0x7fc8bc1457f0 legacy=0x7fc8bc144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 >> v1:172.21.15.64:6789/0 conn(0x7fc8bc13f650 legacy=0x7fc8bc1d6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 monclient: found mon.b 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc144dc0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 5 monclient: authenticate success, global_id 4408 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 10 monclient: _renew_subs 2024-03-29T10:50:40.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc145070 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 10 monclient: _renew_subs 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc1dec00 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.048+0000 7fc8c2756640 1 librados: init done 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8b4001c50 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b9ffb640 4 set_mon_vals no callback set 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc8b4004d00 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.049+0000 7fc8b8ff9640 10 monclient: dump: 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:40.060 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.050+0000 7fc8c2756640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.050+0000 7fc8c2756640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.050+0000 7fc8c2756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:40.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.050+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc898005080 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.051+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fc8b401acd0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fc8b4084ca0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8b8ff9640 10 client.4408.objecter ms_dispatch 0x7fc8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8b8ff9640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8b8ff9640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8b8ff9640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8bb7fe640 10 monclient: get_auth_request con 0x7fc8a406de80 auth_method 0 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.052+0000 7fc8b8ff9640 10 client.4408.objecter ms_handle_connect 0x7fc8a406de80 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.060+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8b4050580 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.060+0000 7fc8b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:40.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.060+0000 7fc8b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.418+0000 7fc8c2756640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:40.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.418+0000 7fc8c2756640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:40.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.418+0000 7fc8c2756640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:50:40.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.418+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc898005330 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.419+0000 7fc8b8ff9640 1 -- 172.21.15.64:0/3038935738 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fc8b40524b0 con 0x7fc8bc0a9d70 2024-03-29T10:50:40.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.419+0000 7fc8b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:40.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.419+0000 7fc8b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:40.401 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:40.401 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 10 librados: watch_flush enter 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 10 librados: watch_flush exit 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 10 client.4408.objecter successfully canceled tick 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 >> v1:172.21.15.174:6816/2473791532 conn(0x7fc8a406de80 legacy=0x7fc8a4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 10 monclient: shutdown 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 >> v1:172.21.15.174:6789/0 conn(0x7fc8bc0a9d70 legacy=0x7fc8bc1d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 shutdown_connections 2024-03-29T10:50:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 >> 172.21.15.64:0/3038935738 conn(0x7fc8bc066a00 msgr2=0x7fc8bc0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 shutdown_connections 2024-03-29T10:50:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 -- 172.21.15.64:0/3038935738 wait complete. 2024-03-29T10:50:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:40.422+0000 7fc8c2756640 1 librados: shutdown 2024-03-29T10:50:40.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:47.225 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.391+0000 7fa33f73e640 10 monclient: get_monmap_and_config 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.391+0000 7fa33f73e640 10 monclient: build_initial_monmap 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.391+0000 7fa33f73e640 10 monclient: monmap: 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:47.392193+0000 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:47.392193+0000 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:47.374 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:47.375 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:47.375 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:47.375 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:47.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.392+0000 7fa33f73e640 1 Processor -- start 2024-03-29T10:50:47.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.392+0000 7fa33f73e640 1 -- start start 2024-03-29T10:50:47.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.392+0000 7fa33f73e640 10 monclient: init 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient(hunting): picked mon.noname-a con 0x7fa3380a9930 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient(hunting): picked mon.noname-c con 0x7fa338149dc0 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient(hunting): picked mon.noname-b con 0x7fa33814ff40 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380aae60 con 0x7fa3380a9930 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380b4310 con 0x7fa338149dc0 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380afdd0 con 0x7fa33814ff40 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.393+0000 7fa33f73e640 10 monclient(hunting): authenticate will time out at 1058.692615s 2024-03-29T10:50:47.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.394+0000 7fa33ccd9640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa338149dc0 0x7fa33814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:50780/0 (socket says 172.21.15.64:50780) 2024-03-29T10:50:47.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.394+0000 7fa33ccd9640 1 -- 172.21.15.64:0/1103112994 learned_addr learned my addr 172.21.15.64:0/1103112994 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:47.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861531921 0 0) 0x7fa3380afdd0 con 0x7fa33814ff40 2024-03-29T10:50:47.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:47.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:47.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:47.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:50:47.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324003520 con 0x7fa33814ff40 2024-03-29T10:50:47.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497236896 0 0) 0x7fa3380b4310 con 0x7fa338149dc0 2024-03-29T10:50:47.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:47.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:47.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:47.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:50:47.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.397+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3380afdd0 con 0x7fa338149dc0 2024-03-29T10:50:47.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405381140 0 0) 0x7fa324003520 con 0x7fa33814ff40 2024-03-29T10:50:47.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 >> v1:172.21.15.174:6789/0 conn(0x7fa338149dc0 legacy=0x7fa33814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 >> v1:172.21.15.64:6789/0 conn(0x7fa3380a9930 legacy=0x7fa338143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 1 monclient: found mon.noname-b 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33814a950 con 0x7fa33814ff40 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa330001a10 con 0x7fa33814ff40 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.398+0000 7fa32ffff640 10 monclient: dump: 2024-03-29T10:50:47.384 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.399+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa330001b70 con 0x7fa33814ff40 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.399+0000 7fa32ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.399+0000 7fa32ffff640 1 -- 172.21.15.64:0/1103112994 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa330001f80 con 0x7fa33814ff40 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.399+0000 7fa32ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:47.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.399+0000 7fa32ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.399+0000 7fa32ffff640 10 monclient: dump: 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.400+0000 7fa33f73e640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.400+0000 7fa33f73e640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:47.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.400+0000 7fa33f73e640 4 set_mon_vals no callback set 2024-03-29T10:50:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.401+0000 7fa33f73e640 10 monclient: shutdown 2024-03-29T10:50:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.401+0000 7fa33f73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.401+0000 7fa33f73e640 1 -- 172.21.15.64:0/1103112994 >> v1:172.21.15.64:6790/0 conn(0x7fa33814ff40 legacy=0x7fa338152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.401+0000 7fa33f73e640 1 -- 172.21.15.64:0/1103112994 shutdown_connections 2024-03-29T10:50:47.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.401+0000 7fa33f73e640 1 -- 172.21.15.64:0/1103112994 >> 172.21.15.64:0/1103112994 conn(0x7fa3380b35e0 msgr2=0x7fa3380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:47.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.402+0000 7fa33f73e640 1 -- 172.21.15.64:0/1103112994 shutdown_connections 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.402+0000 7fa33f73e640 1 -- 172.21.15.64:0/1103112994 wait complete. 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.403+0000 7fa33f73e640 10 monclient: build_initial_monmap 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.403+0000 7fa33f73e640 10 monclient: monmap: 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:47.390 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 1 librados: starting msgr at 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 1 librados: starting objecter 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 1 Processor -- start 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 1 -- start start 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 1 librados: setting wanted keys 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 1 librados: calling monclient init 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.404+0000 7fa33f73e640 10 monclient: init 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:47.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient(hunting): picked mon.noname-b con 0x7fa3380a9930 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient(hunting): picked mon.noname-c con 0x7fa338149dc0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient(hunting): picked mon.noname-a con 0x7fa33814ff40 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3381dc060 con 0x7fa33814ff40 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3381dc860 con 0x7fa3380a9930 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380acd80 con 0x7fa338149dc0 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:47.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.405+0000 7fa33f73e640 10 monclient(hunting): authenticate will time out at 1058.704214s 2024-03-29T10:50:47.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.408+0000 7fa33d4da640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa3380a9930 0x7fa338149330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:51460/0 (socket says 172.21.15.64:51460) 2024-03-29T10:50:47.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.408+0000 7fa33d4da640 1 -- 172.21.15.64:0/84126451 learned_addr learned my addr 172.21.15.64:0/84126451 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:47.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 client.?.objecter ms_handle_connect 0x7fa33814ff40 2024-03-29T10:50:47.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407071642 0 0) 0x7fa3381dc060 con 0x7fa33814ff40 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa31c004eb0 con 0x7fa33814ff40 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310620029 0 0) 0x7fa31c004eb0 con 0x7fa33814ff40 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3381dc060 con 0x7fa33814ff40 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa330001ec0 con 0x7fa33814ff40 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): dump: 2024-03-29T10:50:47.398 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.409+0000 7fa32dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 client.?.objecter ms_handle_connect 0x7fa338149dc0 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:47.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1926362505 0 0) 0x7fa3381dc060 con 0x7fa33814ff40 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 >> v1:172.21.15.64:6790/0 conn(0x7fa338149dc0 legacy=0x7fa3381db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 >> v1:172.21.15.174:6789/0 conn(0x7fa3380a9930 legacy=0x7fa338149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 monclient: found mon.a 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33813eef0 con 0x7fa33814ff40 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 client.?.objecter ms_handle_connect 0x7fa3380a9930 2024-03-29T10:50:47.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 10 monclient: _renew_subs 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33813f9a0 con 0x7fa33814ff40 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 10 monclient: _renew_subs 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33813f710 con 0x7fa33814ff40 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa33f73e640 1 librados: init done 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa330003c20 con 0x7fa33814ff40 2024-03-29T10:50:47.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa330003f30 con 0x7fa33814ff40 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32f7fe640 4 set_mon_vals no callback set 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.410+0000 7fa32dffb640 10 monclient: dump: 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:47.402 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:47.403 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:47.403 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:47.403 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:47.403 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:47.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.415+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fa330004090 con 0x7fa33814ff40 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.416+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fa330083c60 con 0x7fa33814ff40 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.416+0000 7fa32dffb640 10 client.4389.objecter ms_dispatch 0x7fa338000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.416+0000 7fa32dffb640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.416+0000 7fa32dffb640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.416+0000 7fa32dffb640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.417+0000 7fa33f73e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.417+0000 7fa33f73e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:47.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.417+0000 7fa33f73e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:47.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.417+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa318005080 con 0x7fa33814ff40 2024-03-29T10:50:47.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.418+0000 7fa33d4da640 10 monclient: get_auth_request con 0x7fa31c06dbf0 auth_method 0 2024-03-29T10:50:47.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.420+0000 7fa32dffb640 10 client.4389.objecter ms_handle_connect 0x7fa31c06dbf0 2024-03-29T10:50:47.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.425+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa33004f540 con 0x7fa33814ff40 2024-03-29T10:50:47.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.425+0000 7fa32dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:47.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.425+0000 7fa32dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:47.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.752+0000 7fa33f73e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:47.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.752+0000 7fa33f73e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:47.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.752+0000 7fa33f73e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:50:47.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.752+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa318005330 con 0x7fa33814ff40 2024-03-29T10:50:47.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.755+0000 7fa32dffb640 1 -- 172.21.15.64:0/84126451 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fa330051470 con 0x7fa33814ff40 2024-03-29T10:50:47.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.755+0000 7fa32dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:47.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.755+0000 7fa32dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:47.737 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:47.737 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:47.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.759+0000 7fa33f73e640 10 librados: watch_flush enter 2024-03-29T10:50:47.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.759+0000 7fa33f73e640 10 librados: watch_flush exit 2024-03-29T10:50:47.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.759+0000 7fa33f73e640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T10:50:47.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.759+0000 7fa33f73e640 10 client.4389.objecter successfully canceled tick 2024-03-29T10:50:47.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.760+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 >> v1:172.21.15.174:6816/2473791532 conn(0x7fa31c06dbf0 legacy=0x7fa31c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:47.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.760+0000 7fa33f73e640 10 monclient: shutdown 2024-03-29T10:50:47.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:47.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 >> v1:172.21.15.64:6789/0 conn(0x7fa33814ff40 legacy=0x7fa3380ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:47.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 shutdown_connections 2024-03-29T10:50:47.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 >> 172.21.15.64:0/84126451 conn(0x7fa3380b35e0 msgr2=0x7fa338140ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:47.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 shutdown_connections 2024-03-29T10:50:47.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 1 -- 172.21.15.64:0/84126451 wait complete. 2024-03-29T10:50:47.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:47.761+0000 7fa33f73e640 1 librados: shutdown 2024-03-29T10:50:47.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:54.566 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:54.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.731+0000 7fea88f94640 10 monclient: get_monmap_and_config 2024-03-29T10:50:54.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.731+0000 7fea88f94640 10 monclient: build_initial_monmap 2024-03-29T10:50:54.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.731+0000 7fea88f94640 10 monclient: monmap: 2024-03-29T10:50:54.714 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:54.714 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:50:54.732315+0000 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:50:54.732315+0000 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.732+0000 7fea88f94640 1 Processor -- start 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.732+0000 7fea88f94640 1 -- start start 2024-03-29T10:50:54.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.732+0000 7fea88f94640 10 monclient: init 2024-03-29T10:50:54.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:54.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:54.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient(hunting): picked mon.noname-b con 0x7fea7c006a70 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:54.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient(hunting): picked mon.noname-c con 0x7fea7c0e33e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:54.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient(hunting): picked mon.noname-a con 0x7fea7c02b630 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:54.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c035480 con 0x7fea7c02b630 2024-03-29T10:50:54.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c02c9c0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c02cca0 con 0x7fea7c006a70 2024-03-29T10:50:54.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:54.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.733+0000 7fea88f94640 10 monclient(hunting): authenticate will time out at 1066.032576s 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.734+0000 7fea82ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fea7c0e33e0 0x7fea7c0e9220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:51460/0 (socket says 172.21.15.64:51460) 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.734+0000 7fea82ffd640 1 -- 172.21.15.64:0/1728421378 learned_addr learned my addr 172.21.15.64:0/1728421378 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272113446 0 0) 0x7fea7c02c9c0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74003520 con 0x7fea7c0e33e0 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 422214297 0 0) 0x7fea74003520 con 0x7fea7c0e33e0 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.735+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 >> v1:172.21.15.64:6790/0 conn(0x7fea7c006a70 legacy=0x7fea7c0e2cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:54.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.736+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 >> v1:172.21.15.64:6789/0 conn(0x7fea7c02b630 legacy=0x7fea7c0e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:54.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:54.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 1 monclient: found mon.noname-c 2024-03-29T10:50:54.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:54.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c02ab40 con 0x7fea7c0e33e0 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea70001670 con 0x7fea7c0e33e0 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.739+0000 7fea827fc640 10 monclient: dump: 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:54.724 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea88f94640 5 monclient: authenticate success, global_id 4414 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea88f94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea700018d0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea827fc640 1 -- 172.21.15.64:0/1728421378 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea70001d20 con 0x7fea7c0e33e0 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea827fc640 10 monclient: dump: 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:54.725 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea88f94640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.740+0000 7fea88f94640 4 set_mon_vals no callback set 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 10 monclient: shutdown 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 1 -- 172.21.15.64:0/1728421378 >> v1:172.21.15.174:6789/0 conn(0x7fea7c0e33e0 legacy=0x7fea7c0e9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:54.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 1 -- 172.21.15.64:0/1728421378 shutdown_connections 2024-03-29T10:50:54.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 1 -- 172.21.15.64:0/1728421378 >> 172.21.15.64:0/1728421378 conn(0x7fea7c0058f0 msgr2=0x7fea7c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:54.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 1 -- 172.21.15.64:0/1728421378 shutdown_connections 2024-03-29T10:50:54.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.741+0000 7fea88f94640 1 -- 172.21.15.64:0/1728421378 wait complete. 2024-03-29T10:50:54.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.742+0000 7fea88f94640 10 monclient: build_initial_monmap 2024-03-29T10:50:54.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.742+0000 7fea88f94640 10 monclient: monmap: 2024-03-29T10:50:54.728 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:50:54.728 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 1 librados: starting msgr at 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 1 librados: starting objecter 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 1 Processor -- start 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 1 -- start start 2024-03-29T10:50:54.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 1 librados: setting wanted keys 2024-03-29T10:50:54.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 1 librados: calling monclient init 2024-03-29T10:50:54.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.743+0000 7fea88f94640 10 monclient: init 2024-03-29T10:50:54.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.744+0000 7fea88f94640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:54.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.744+0000 7fea88f94640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:54.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.744+0000 7fea88f94640 10 monclient(hunting): picked mon.noname-a con 0x7fea7c006a70 addr v1:172.21.15.64:6789/0 2024-03-29T10:50:54.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.744+0000 7fea88f94640 10 monclient(hunting): picked mon.noname-b con 0x7fea7c02b630 addr v1:172.21.15.174:6789/0 2024-03-29T10:50:54.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea88f94640 10 monclient(hunting): picked mon.noname-c con 0x7fea7c0e33e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:50:54.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea88f94640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c17ff90 con 0x7fea7c006a70 2024-03-29T10:50:54.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea88f94640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0dc1f0 con 0x7fea7c02b630 2024-03-29T10:50:54.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea88f94640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7c0dc4d0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea88f94640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:54.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea88f94640 10 monclient(hunting): authenticate will time out at 1066.046336s 2024-03-29T10:50:54.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea82ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fea7c02b630 0x7fea7c17f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:52030/0 (socket says 172.21.15.64:52030) 2024-03-29T10:50:54.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea82ffd640 1 -- 172.21.15.64:0/1934485001 learned_addr learned my addr 172.21.15.64:0/1934485001 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:50:54.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea6ffff640 10 client.?.objecter ms_handle_connect 0x7fea7c02b630 2024-03-29T10:50:54.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.747+0000 7fea6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.750+0000 7fea6ffff640 10 client.?.objecter ms_handle_connect 0x7fea7c0e33e0 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.750+0000 7fea6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.750+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998685061 0 0) 0x7fea7c0dc1f0 con 0x7fea7c02b630 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.750+0000 7fea6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.750+0000 7fea6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.750+0000 7fea6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea68004eb0 con 0x7fea7c02b630 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333995459 0 0) 0x7fea7c0dc4d0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:54.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c0dc1f0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.751+0000 7fea6ffff640 10 client.?.objecter ms_handle_connect 0x7fea7c006a70 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125068254 0 0) 0x7fea7c17ff90 con 0x7fea7c006a70 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c0dc4d0 con 0x7fea7c006a70 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2109538406 0 0) 0x7fea7c0dc1f0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7c17ff90 con 0x7fea7c0e33e0 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea8404fdd0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.752+0000 7fea6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 10 monclient(hunting): dump: 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:54.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896765513 0 0) 0x7fea7c0dc4d0 con 0x7fea7c006a70 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7c0dc1f0 con 0x7fea7c006a70 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2481336155 0 0) 0x7fea68004eb0 con 0x7fea7c02b630 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7c0dc4d0 con 0x7fea7c02b630 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3135232034 0 0) 0x7fea7c17ff90 con 0x7fea7c0e33e0 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 >> v1:172.21.15.174:6789/0 conn(0x7fea7c02b630 legacy=0x7fea7c17f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 >> v1:172.21.15.64:6789/0 conn(0x7fea7c006a70 legacy=0x7fea7c18ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 monclient: found mon.c 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c16f8d0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:54.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 10 monclient: _renew_subs 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7c16fdf0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 10 monclient: _renew_subs 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7c18f700 con 0x7fea7c0e33e0 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.753+0000 7fea88f94640 1 librados: init done 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea8404f400 con 0x7fea7c0e33e0 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fea84075f10 con 0x7fea7c0e33e0 2024-03-29T10:50:54.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 10 monclient: dump: 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:50:54.744 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea81ffb640 4 set_mon_vals no callback set 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea88f94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.754+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fea84067bf0 con 0x7fea7c0e33e0 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.756+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fea840ed060 con 0x7fea7c0e33e0 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.756+0000 7fea6ffff640 10 client.4424.objecter ms_dispatch 0x7fea84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.756+0000 7fea6ffff640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.756+0000 7fea6ffff640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.756+0000 7fea6ffff640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T10:50:54.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.757+0000 7fea88f94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:54.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.757+0000 7fea88f94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:50:54.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.757+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea64005080 con 0x7fea7c0e33e0 2024-03-29T10:50:54.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.757+0000 7fea837fe640 10 monclient: get_auth_request con 0x7fea6806dde0 auth_method 0 2024-03-29T10:50:54.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.758+0000 7fea6ffff640 10 client.4424.objecter ms_handle_connect 0x7fea6806dde0 2024-03-29T10:50:54.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.765+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea840b8940 con 0x7fea7c0e33e0 2024-03-29T10:50:54.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.765+0000 7fea6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:54.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:54.765+0000 7fea6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.109+0000 7fea88f94640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.109+0000 7fea88f94640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.109+0000 7fea88f94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:50:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.109+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea64005330 con 0x7fea7c0e33e0 2024-03-29T10:50:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.110+0000 7fea6ffff640 1 -- 172.21.15.64:0/1934485001 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fea840ba870 con 0x7fea7c0e33e0 2024-03-29T10:50:55.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.110+0000 7fea6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:55.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.110+0000 7fea6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:55.092 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:50:55.092 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:55.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.114+0000 7fea88f94640 10 librados: watch_flush enter 2024-03-29T10:50:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 10 librados: watch_flush exit 2024-03-29T10:50:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T10:50:55.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 10 client.4424.objecter successfully canceled tick 2024-03-29T10:50:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 >> v1:172.21.15.174:6816/2473791532 conn(0x7fea6806dde0 legacy=0x7fea680702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 10 monclient: shutdown 2024-03-29T10:50:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.115+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 >> v1:172.21.15.64:6790/0 conn(0x7fea7c0e33e0 legacy=0x7fea7c17f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.116+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 shutdown_connections 2024-03-29T10:50:55.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.116+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 >> 172.21.15.64:0/1934485001 conn(0x7fea7c0058f0 msgr2=0x7fea7c0e2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:55.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.118+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 shutdown_connections 2024-03-29T10:50:55.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.118+0000 7fea88f94640 1 -- 172.21.15.64:0/1934485001 wait complete. 2024-03-29T10:50:55.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:50:55.118+0000 7fea88f94640 1 librados: shutdown 2024-03-29T10:50:55.117 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:01.926 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:02.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.101+0000 7f0303053640 10 monclient: get_monmap_and_config 2024-03-29T10:51:02.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.101+0000 7f0303053640 10 monclient: build_initial_monmap 2024-03-29T10:51:02.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.101+0000 7f0303053640 10 monclient: monmap: 2024-03-29T10:51:02.084 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:02.084 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:02.084 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:02.102708+0000 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:02.102708+0000 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:02.085 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:02.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.104+0000 7f0303053640 1 Processor -- start 2024-03-29T10:51:02.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.106+0000 7f0303053640 1 -- start start 2024-03-29T10:51:02.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.106+0000 7f0303053640 10 monclient: init 2024-03-29T10:51:02.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:02.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:02.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient(hunting): picked mon.noname-c con 0x7f02fc0aaee0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:02.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient(hunting): picked mon.noname-a con 0x7f02fc0aa500 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient(hunting): picked mon.noname-b con 0x7f02fc149800 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc0aded0 con 0x7f02fc0aa500 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc0ad7e0 con 0x7f02fc0aaee0 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc0ac350 con 0x7f02fc149800 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.107+0000 7f0303053640 10 monclient(hunting): authenticate will time out at 1073.406828s 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.111+0000 7f0302852640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f02fc149800 0x7f02fc151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45505/0 (socket says 172.21.15.64:45505) 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.111+0000 7f0302852640 1 -- 172.21.15.64:0/700025240 learned_addr learned my addr 172.21.15.64:0/700025240 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.111+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234933525 0 0) 0x7f02fc0ac350 con 0x7f02fc149800 2024-03-29T10:51:02.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 10 monclient(hunting): _negotiate 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e8003520 con 0x7f02fc149800 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836687404 0 0) 0x7f02e8003520 con 0x7f02fc149800 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 >> v1:172.21.15.174:6789/0 conn(0x7f02fc0aaee0 legacy=0x7f02fc14a140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:51:02.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.112+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 >> v1:172.21.15.64:6789/0 conn(0x7f02fc0aa500 legacy=0x7f02fc0ad0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: _finish_hunting 0 2024-03-29T10:51:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 1 monclient: found mon.noname-b 2024-03-29T10:51:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:51:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc0ac630 con 0x7f02fc149800 2024-03-29T10:51:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: _finish_auth 0 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: _check_auth_tickets 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f0303053640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f0303053640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f02f4001a10 con 0x7f02fc149800 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.114+0000 7f030104f640 10 monclient: dump: 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.099 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:02.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.115+0000 7f0303053640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.117+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f4001ca0 con 0x7f02fc149800 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.117+0000 7f030104f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.117+0000 7f0303053640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.117+0000 7f0303053640 4 set_mon_vals no callback set 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f030104f640 1 -- 172.21.15.64:0/700025240 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f02f4002030 con 0x7f02fc149800 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f030104f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f030104f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f030104f640 10 monclient: dump: 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.101 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f0303053640 10 monclient: shutdown 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f0303053640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f0303053640 1 -- 172.21.15.64:0/700025240 >> v1:172.21.15.64:6790/0 conn(0x7f02fc149800 legacy=0x7f02fc151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f0303053640 1 -- 172.21.15.64:0/700025240 shutdown_connections 2024-03-29T10:51:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.118+0000 7f0303053640 1 -- 172.21.15.64:0/700025240 >> 172.21.15.64:0/700025240 conn(0x7f02fc0b3a30 msgr2=0x7f02fc0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:02.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.119+0000 7f0303053640 1 -- 172.21.15.64:0/700025240 shutdown_connections 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.121+0000 7f0303053640 1 -- 172.21.15.64:0/700025240 wait complete. 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.122+0000 7f0303053640 10 monclient: build_initial_monmap 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.122+0000 7f0303053640 10 monclient: monmap: 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:02.104 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.122+0000 7f0303053640 1 librados: starting msgr at 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.122+0000 7f0303053640 1 librados: starting objecter 2024-03-29T10:51:02.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 1 Processor -- start 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 1 -- start start 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 1 librados: setting wanted keys 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 1 librados: calling monclient init 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 10 monclient: init 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:02.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 10 monclient(hunting): picked mon.noname-c con 0x7f02fc0aa500 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 10 monclient(hunting): picked mon.noname-b con 0x7f02fc0aaee0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.123+0000 7f0303053640 10 monclient(hunting): picked mon.noname-a con 0x7f02fc149800 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.124+0000 7f0303053640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc06f880 con 0x7f02fc149800 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.124+0000 7f0303053640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc0708b0 con 0x7f02fc0aaee0 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.124+0000 7f0303053640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc070b90 con 0x7f02fc0aa500 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.124+0000 7f0303053640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:02.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.124+0000 7f0303053640 10 monclient(hunting): authenticate will time out at 1073.422887s 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.127+0000 7f0302852640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f02fc149800 0x7f02fc1e7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45477/0 (socket says 172.21.15.64:45477) 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.127+0000 7f0302852640 1 -- 172.21.15.64:0/1033419669 learned_addr learned my addr 172.21.15.64:0/1033419669 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.127+0000 7f02f2ffd640 10 client.?.objecter ms_handle_connect 0x7f02fc149800 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.127+0000 7f02f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.129+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341396519 0 0) 0x7f02fc06f880 con 0x7f02fc149800 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.129+0000 7f02f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.129+0000 7f02f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.129+0000 7f02f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.129+0000 7f02f2ffd640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:51:02.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.129+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0004eb0 con 0x7f02fc149800 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 client.?.objecter ms_handle_connect 0x7f02fc0aaee0 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031846036 0 0) 0x7f02e0004eb0 con 0x7f02fc149800 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02fc06f880 con 0x7f02fc149800 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f02f40018d0 con 0x7f02fc149800 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:02.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:02.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163175705 0 0) 0x7f02fc0708b0 con 0x7f02fc0aaee0 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0004eb0 con 0x7f02fc0aaee0 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3580635497 0 0) 0x7f02fc06f880 con 0x7f02fc149800 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.131+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 >> v1:172.21.15.64:6790/0 conn(0x7f02fc0aa500 legacy=0x7f02fc1430c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 >> v1:172.21.15.174:6789/0 conn(0x7f02fc0aaee0 legacy=0x7f02fc1e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 1 monclient: found mon.a 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:02.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc06e7a0 con 0x7f02fc149800 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f02f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f0303053640 5 monclient: authenticate success, global_id 4398 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f0303053640 10 monclient: _renew_subs 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f0303053640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.132+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc06f340 con 0x7f02fc149800 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f0303053640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:02.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f0303053640 10 monclient: _renew_subs 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f0303053640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc146080 con 0x7f02fc149800 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f0303053640 1 librados: init done 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f4004f70 con 0x7f02fc149800 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f02f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f02f4005280 con 0x7f02fc149800 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f02f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f02f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f02f2ffd640 10 monclient: dump: 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.119 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.133+0000 7f030084e640 4 set_mon_vals no callback set 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.134+0000 7f0303053640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:02.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.134+0000 7f0303053640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.134+0000 7f0303053640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.134+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e4005080 con 0x7f02fc149800 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.135+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f02f401b250 con 0x7f02fc149800 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.136+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f02f4084f80 con 0x7f02fc149800 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.136+0000 7f02f2ffd640 10 client.4398.objecter ms_dispatch 0x7f02fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.136+0000 7f02f2ffd640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.136+0000 7f02f2ffd640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:02.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.136+0000 7f02f2ffd640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T10:51:02.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.142+0000 7f0302051640 10 monclient: get_auth_request con 0x7f02e006db40 auth_method 0 2024-03-29T10:51:02.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.143+0000 7f02f2ffd640 10 client.4398.objecter ms_handle_connect 0x7f02e006db40 2024-03-29T10:51:02.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.145+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02f4050860 con 0x7f02fc149800 2024-03-29T10:51:02.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.145+0000 7f02f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:02.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.145+0000 7f02f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:02.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.476+0000 7f0303053640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:02.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.476+0000 7f0303053640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:02.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.476+0000 7f0303053640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:02.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.476+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e4005330 con 0x7f02fc149800 2024-03-29T10:51:02.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.478+0000 7f02f2ffd640 1 -- 172.21.15.64:0/1033419669 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f02f4052790 con 0x7f02fc149800 2024-03-29T10:51:02.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.478+0000 7f02f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:02.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.478+0000 7f02f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:02.460 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:02.461 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:02.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.482+0000 7f0303053640 10 librados: watch_flush enter 2024-03-29T10:51:02.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 10 librados: watch_flush exit 2024-03-29T10:51:02.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T10:51:02.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 10 client.4398.objecter successfully canceled tick 2024-03-29T10:51:02.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 >> v1:172.21.15.174:6816/2473791532 conn(0x7f02e006db40 legacy=0x7f02e0070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:02.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 10 monclient: shutdown 2024-03-29T10:51:02.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:02.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.483+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 >> v1:172.21.15.64:6789/0 conn(0x7f02fc149800 legacy=0x7f02fc1e7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:02.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.485+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 shutdown_connections 2024-03-29T10:51:02.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.485+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 >> 172.21.15.64:0/1033419669 conn(0x7f02fc0b3a30 msgr2=0x7f02fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:02.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.485+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 shutdown_connections 2024-03-29T10:51:02.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.485+0000 7f0303053640 1 -- 172.21.15.64:0/1033419669 wait complete. 2024-03-29T10:51:02.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:02.485+0000 7f0303053640 1 librados: shutdown 2024-03-29T10:51:02.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:09.290 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:09.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.462+0000 7f634f063640 10 monclient: get_monmap_and_config 2024-03-29T10:51:09.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.462+0000 7f634f063640 10 monclient: build_initial_monmap 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.462+0000 7f634f063640 10 monclient: monmap: 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:09.462996+0000 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:09.462996+0000 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:09.444 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:09.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.464+0000 7f634f063640 1 Processor -- start 2024-03-29T10:51:09.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.465+0000 7f634f063640 1 -- start start 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.465+0000 7f634f063640 10 monclient: init 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient(hunting): picked mon.noname-c con 0x7f63480b2800 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient(hunting): picked mon.noname-b con 0x7f63480a9c00 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient(hunting): picked mon.noname-a con 0x7f63480b38c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6348141eb0 con 0x7f63480b38c0 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63480b22d0 con 0x7f63480b2800 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6348142d30 con 0x7f63480a9c00 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:09.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.466+0000 7f634f063640 10 monclient(hunting): authenticate will time out at 1080.765254s 2024-03-29T10:51:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.469+0000 7f634e862640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f63480b38c0 0x7f634814d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57709/0 (socket says 172.21.15.64:57709) 2024-03-29T10:51:09.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.470+0000 7f634e862640 1 -- 172.21.15.64:0/2545205925 learned_addr learned my addr 172.21.15.64:0/2545205925 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:09.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.473+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617448807 0 0) 0x7f6348141eb0 con 0x7f63480b38c0 2024-03-29T10:51:09.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.473+0000 7f634d05f640 10 monclient(hunting): _negotiate 2024-03-29T10:51:09.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.473+0000 7f634d05f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:09.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.473+0000 7f634d05f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:09.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.473+0000 7f634d05f640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:51:09.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.473+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6334003520 con 0x7f63480b38c0 2024-03-29T10:51:09.456 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.474+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3779739231 0 0) 0x7f6334003520 con 0x7f63480b38c0 2024-03-29T10:51:09.456 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 >> v1:172.21.15.64:6790/0 conn(0x7f63480a9c00 legacy=0x7f634813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.456 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 >> v1:172.21.15.174:6789/0 conn(0x7f63480b2800 legacy=0x7f6348142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 10 monclient: _finish_hunting 0 2024-03-29T10:51:09.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 1 monclient: found mon.noname-a 2024-03-29T10:51:09.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:09.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:51:09.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.475+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63480aaf90 con 0x7f63480b38c0 2024-03-29T10:51:09.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.476+0000 7f634d05f640 10 monclient: _finish_auth 0 2024-03-29T10:51:09.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.476+0000 7f634d05f640 10 monclient: _check_auth_tickets 2024-03-29T10:51:09.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.476+0000 7f634d05f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:09.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.477+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6340001a10 con 0x7f63480b38c0 2024-03-29T10:51:09.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.477+0000 7f634d05f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:09.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.477+0000 7f634d05f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:09.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.477+0000 7f634d05f640 10 monclient: dump: 2024-03-29T10:51:09.460 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:09.460 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.477+0000 7f634f063640 5 monclient: authenticate success, global_id 4401 2024-03-29T10:51:09.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.477+0000 7f634f063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:09.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.479+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6340001c30 con 0x7f63480b38c0 2024-03-29T10:51:09.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.479+0000 7f634d05f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:09.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.479+0000 7f634d05f640 1 -- 172.21.15.64:0/2545205925 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6340001f40 con 0x7f63480b38c0 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.479+0000 7f634d05f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.479+0000 7f634d05f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.479+0000 7f634d05f640 10 monclient: dump: 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:09.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.480+0000 7f634f063640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:09.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.480+0000 7f634f063640 4 set_mon_vals no callback set 2024-03-29T10:51:09.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.480+0000 7f634f063640 10 monclient: shutdown 2024-03-29T10:51:09.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.480+0000 7f634f063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:09.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.480+0000 7f634f063640 1 -- 172.21.15.64:0/2545205925 >> v1:172.21.15.64:6789/0 conn(0x7f63480b38c0 legacy=0x7f634814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.481+0000 7f634f063640 1 -- 172.21.15.64:0/2545205925 shutdown_connections 2024-03-29T10:51:09.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.481+0000 7f634f063640 1 -- 172.21.15.64:0/2545205925 >> 172.21.15.64:0/2545205925 conn(0x7f6348066a00 msgr2=0x7f6348066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.482+0000 7f634f063640 1 -- 172.21.15.64:0/2545205925 shutdown_connections 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.483+0000 7f634f063640 1 -- 172.21.15.64:0/2545205925 wait complete. 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.484+0000 7f634f063640 10 monclient: build_initial_monmap 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.484+0000 7f634f063640 10 monclient: monmap: 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:09.465 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:09.466 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:09.466 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:09.466 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:09.466 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.484+0000 7f634f063640 1 librados: starting msgr at 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.484+0000 7f634f063640 1 librados: starting objecter 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.485+0000 7f634f063640 1 Processor -- start 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.485+0000 7f634f063640 1 -- start start 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.485+0000 7f634f063640 1 librados: setting wanted keys 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.485+0000 7f634f063640 1 librados: calling monclient init 2024-03-29T10:51:09.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.485+0000 7f634f063640 10 monclient: init 2024-03-29T10:51:09.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.486+0000 7f634f063640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:09.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.486+0000 7f634f063640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:09.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.489+0000 7f634f063640 10 monclient(hunting): picked mon.noname-b con 0x7f63480a9c00 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:09.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.490+0000 7f634f063640 10 monclient(hunting): picked mon.noname-c con 0x7f63480b2800 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:09.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.490+0000 7f634e061640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f63480a9c00 0x7f63480add70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:40151/0 (socket says 172.21.15.64:40151) 2024-03-29T10:51:09.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.490+0000 7f634e061640 1 -- 172.21.15.64:0/3700319029 learned_addr learned my addr 172.21.15.64:0/3700319029 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:09.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f633effd640 10 client.?.objecter ms_handle_connect 0x7f63480a9c00 2024-03-29T10:51:09.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f633effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:09.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f634f063640 10 monclient(hunting): picked mon.noname-a con 0x7f63480b38c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:09.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63480b01d0 con 0x7f63480b38c0 2024-03-29T10:51:09.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63481d70e0 con 0x7f63480a9c00 2024-03-29T10:51:09.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63480b1590 con 0x7f63480b2800 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f634f063640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.491+0000 7f634f063640 10 monclient(hunting): authenticate will time out at 1080.790190s 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.492+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407343908 0 0) 0x7f63481d70e0 con 0x7f63480a9c00 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.492+0000 7f633effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.492+0000 7f633effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.492+0000 7f633effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.492+0000 7f633effd640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:51:09.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.492+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c004eb0 con 0x7f63480a9c00 2024-03-29T10:51:09.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074913816 0 0) 0x7f632c004eb0 con 0x7f63480a9c00 2024-03-29T10:51:09.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63481d70e0 con 0x7f63480a9c00 2024-03-29T10:51:09.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6344001a30 con 0x7f63480a9c00 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 monclient(hunting): dump: 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:09.478 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 client.?.objecter ms_handle_connect 0x7f63480b38c0 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 client.?.objecter ms_handle_connect 0x7f63480b2800 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.493+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3104018094 0 0) 0x7f63481d70e0 con 0x7f63480a9c00 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 >> v1:172.21.15.64:6790/0 conn(0x7f63480b2800 legacy=0x7f63480aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 >> v1:172.21.15.64:6789/0 conn(0x7f63480b38c0 legacy=0x7f63481d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 1 monclient: found mon.b 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63481d7d10 con 0x7f63480a9c00 2024-03-29T10:51:09.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 10 monclient: _finish_auth 0 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f633effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f634f063640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.494+0000 7f634f063640 10 monclient: _renew_subs 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63481e2960 con 0x7f63480a9c00 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 10 monclient: _renew_subs 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63481e2cd0 con 0x7f63480a9c00 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634f063640 1 librados: init done 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6344001b90 con 0x7f63480a9c00 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f633effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:09.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6344003a60 con 0x7f63480a9c00 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f633effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f633effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f633effd640 10 monclient: dump: 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:09.481 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.495+0000 7f634c85e640 4 set_mon_vals no callback set 2024-03-29T10:51:09.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.497+0000 7f634f063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.497+0000 7f634f063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.497+0000 7f634f063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.497+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6330005080 con 0x7f63480a9c00 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.497+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f6344003bc0 con 0x7f63480a9c00 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.498+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f6344084ba0 con 0x7f63480a9c00 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.498+0000 7f633effd640 10 client.4426.objecter ms_dispatch 0x7f6348000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.498+0000 7f633effd640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.498+0000 7f633effd640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.498+0000 7f633effd640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.498+0000 7f634d860640 10 monclient: get_auth_request con 0x7f632c06db80 auth_method 0 2024-03-29T10:51:09.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.500+0000 7f633effd640 10 client.4426.objecter ms_handle_connect 0x7f632c06db80 2024-03-29T10:51:09.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.506+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6344050480 con 0x7f63480a9c00 2024-03-29T10:51:09.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.506+0000 7f633effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:09.487 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.506+0000 7f633effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:09.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.836+0000 7f634f063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:09.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.836+0000 7f634f063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:09.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.836+0000 7f634f063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:09.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.836+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6330005330 con 0x7f63480a9c00 2024-03-29T10:51:09.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.840+0000 7f633effd640 1 -- 172.21.15.64:0/3700319029 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f63440523b0 con 0x7f63480a9c00 2024-03-29T10:51:09.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.840+0000 7f633effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:09.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.840+0000 7f633effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:09.823 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:09.823 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:09.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.844+0000 7f634f063640 10 librados: watch_flush enter 2024-03-29T10:51:09.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 10 librados: watch_flush exit 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 10 client.4426.objecter successfully canceled tick 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 >> v1:172.21.15.174:6816/2473791532 conn(0x7f632c06db80 legacy=0x7f632c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 10 monclient: shutdown 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 >> v1:172.21.15.174:6789/0 conn(0x7f63480a9c00 legacy=0x7f63480add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 shutdown_connections 2024-03-29T10:51:09.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.846+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 >> 172.21.15.64:0/3700319029 conn(0x7f6348066a00 msgr2=0x7f63480b70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:09.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.848+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 shutdown_connections 2024-03-29T10:51:09.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.848+0000 7f634f063640 1 -- 172.21.15.64:0/3700319029 wait complete. 2024-03-29T10:51:09.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:09.848+0000 7f634f063640 1 librados: shutdown 2024-03-29T10:51:09.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:16.655 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:16.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.821+0000 7fa977ba8640 10 monclient: get_monmap_and_config 2024-03-29T10:51:16.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.822+0000 7fa977ba8640 10 monclient: build_initial_monmap 2024-03-29T10:51:16.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.822+0000 7fa977ba8640 10 monclient: monmap: 2024-03-29T10:51:16.804 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:16.823861+0000 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:16.823861+0000 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:16.805 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:16.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.823+0000 7fa977ba8640 1 Processor -- start 2024-03-29T10:51:16.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.824+0000 7fa977ba8640 1 -- start start 2024-03-29T10:51:16.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.824+0000 7fa977ba8640 10 monclient: init 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient(hunting): picked mon.noname-c con 0x7fa9701457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient(hunting): picked mon.noname-b con 0x7fa97013f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient(hunting): picked mon.noname-a con 0x7fa9700a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9700aaef0 con 0x7fa9700a9d70 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa970145bd0 con 0x7fa9701457f0 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa970145eb0 con 0x7fa97013f650 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.825+0000 7fa977ba8640 10 monclient(hunting): authenticate will time out at 1088.124958s 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.828+0000 7fa976145640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa9700a9d70 0x7fa97014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37753/0 (socket says 172.21.15.64:37753) 2024-03-29T10:51:16.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.828+0000 7fa976145640 1 -- 172.21.15.64:0/2948921815 learned_addr learned my addr 172.21.15.64:0/2948921815 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438343108 0 0) 0x7fa9700aaef0 con 0x7fa9700a9d70 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 10 monclient(hunting): _negotiate 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa95c003520 con 0x7fa9700a9d70 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2235508526 0 0) 0x7fa95c003520 con 0x7fa9700a9d70 2024-03-29T10:51:16.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.829+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 >> v1:172.21.15.64:6790/0 conn(0x7fa97013f650 legacy=0x7fa9700b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:16.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 >> v1:172.21.15.174:6789/0 conn(0x7fa9701457f0 legacy=0x7fa97013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:16.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: _finish_hunting 0 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 1 monclient: found mon.noname-a 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa970146340 con 0x7fa9700a9d70 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: _finish_auth 0 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: _check_auth_tickets 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa968001a10 con 0x7fa9700a9d70 2024-03-29T10:51:16.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: dump: 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa968001be0 con 0x7fa9700a9d70 2024-03-29T10:51:16.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 1 -- 172.21.15.64:0/2948921815 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa968001ff0 con 0x7fa9700a9d70 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa974942640 10 monclient: dump: 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:16.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa977ba8640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:51:16.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa977ba8640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:16.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.833+0000 7fa977ba8640 4 set_mon_vals no callback set 2024-03-29T10:51:16.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.834+0000 7fa977ba8640 10 monclient: shutdown 2024-03-29T10:51:16.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.834+0000 7fa977ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:16.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.834+0000 7fa977ba8640 1 -- 172.21.15.64:0/2948921815 >> v1:172.21.15.64:6789/0 conn(0x7fa9700a9d70 legacy=0x7fa97014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:16.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.834+0000 7fa977ba8640 1 -- 172.21.15.64:0/2948921815 shutdown_connections 2024-03-29T10:51:16.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.834+0000 7fa977ba8640 1 -- 172.21.15.64:0/2948921815 >> 172.21.15.64:0/2948921815 conn(0x7fa970066a00 msgr2=0x7fa970066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:16.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.835+0000 7fa977ba8640 1 -- 172.21.15.64:0/2948921815 shutdown_connections 2024-03-29T10:51:16.820 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.835+0000 7fa977ba8640 1 -- 172.21.15.64:0/2948921815 wait complete. 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 10 monclient: build_initial_monmap 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 10 monclient: monmap: 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 1 librados: starting msgr at 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 1 librados: starting objecter 2024-03-29T10:51:16.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 1 Processor -- start 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 1 -- start start 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 1 librados: setting wanted keys 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 1 librados: calling monclient init 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.836+0000 7fa977ba8640 10 monclient: init 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient(hunting): picked mon.noname-c con 0x7fa9700a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient(hunting): picked mon.noname-b con 0x7fa97013f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient(hunting): picked mon.noname-a con 0x7fa9701457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa970141520 con 0x7fa9701457f0 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9701452f0 con 0x7fa97013f650 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa970144dd0 con 0x7fa9700a9d70 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:16.822 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.837+0000 7fa977ba8640 10 monclient(hunting): authenticate will time out at 1088.137550s 2024-03-29T10:51:16.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa975944640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fa9700a9d70 0x7fa970133120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39921/0 (socket says 172.21.15.64:39921) 2024-03-29T10:51:16.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa975944640 1 -- 172.21.15.64:0/1325867660 learned_addr learned my addr 172.21.15.64:0/1325867660 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:16.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 10 client.?.objecter ms_handle_connect 0x7fa9700a9d70 2024-03-29T10:51:16.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:16.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605832083 0 0) 0x7fa970144dd0 con 0x7fa9700a9d70 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.842+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa954004eb0 con 0x7fa9700a9d70 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1705451546 0 0) 0x7fa954004eb0 con 0x7fa9700a9d70 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa954005300 con 0x7fa9700a9d70 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa96c001790 con 0x7fa9700a9d70 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:16.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): dump: 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:16.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 client.?.objecter ms_handle_connect 0x7fa9701457f0 2024-03-29T10:51:16.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:16.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917379524 0 0) 0x7fa970141520 con 0x7fa9701457f0 2024-03-29T10:51:16.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:16.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.843+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9540058a0 con 0x7fa9701457f0 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 10 client.?.objecter ms_handle_connect 0x7fa97013f650 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1171511210 0 0) 0x7fa954005300 con 0x7fa9700a9d70 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 >> v1:172.21.15.174:6789/0 conn(0x7fa97013f650 legacy=0x7fa97012e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 >> v1:172.21.15.64:6789/0 conn(0x7fa9701457f0 legacy=0x7fa97012efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:16.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 1 monclient: found mon.c 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97012f6f0 con 0x7fa9700a9d70 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.846+0000 7fa9667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:16.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.847+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa96c001d60 con 0x7fa9700a9d70 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.847+0000 7fa9667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.847+0000 7fa967fff640 4 set_mon_vals no callback set 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fa96c001790 con 0x7fa9700a9d70 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa9667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa9667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa9667fc640 10 monclient: dump: 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:16.831 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 10 monclient: _renew_subs 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97012f9a0 con 0x7fa9700a9d70 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 10 monclient: _renew_subs 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97013ad10 con 0x7fa9700a9d70 2024-03-29T10:51:16.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.848+0000 7fa977ba8640 1 librados: init done 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.849+0000 7fa977ba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.849+0000 7fa977ba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.849+0000 7fa977ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.849+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa970075f10 con 0x7fa9700a9d70 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.850+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fa96c01a750 con 0x7fa9700a9d70 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.851+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fa96c084380 con 0x7fa9700a9d70 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.851+0000 7fa9667fc640 10 client.4439.objecter ms_dispatch 0x7fa970000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.851+0000 7fa9667fc640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.851+0000 7fa9667fc640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.851+0000 7fa9667fc640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T10:51:16.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.851+0000 7fa975143640 10 monclient: get_auth_request con 0x7fa95406e4c0 auth_method 0 2024-03-29T10:51:16.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.853+0000 7fa9667fc640 10 client.4439.objecter ms_handle_connect 0x7fa95406e4c0 2024-03-29T10:51:16.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.858+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa96c04fbd0 con 0x7fa9700a9d70 2024-03-29T10:51:16.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.858+0000 7fa9667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:16.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:16.858+0000 7fa9667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:17.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.181+0000 7fa977ba8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:17.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.181+0000 7fa977ba8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:17.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.181+0000 7fa977ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:17.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.181+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa970144f60 con 0x7fa9700a9d70 2024-03-29T10:51:17.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.182+0000 7fa9667fc640 1 -- 172.21.15.64:0/1325867660 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fa96c04fbd0 con 0x7fa9700a9d70 2024-03-29T10:51:17.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.182+0000 7fa9667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:17.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.182+0000 7fa9667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:17.165 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:17.165 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:17.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.185+0000 7fa977ba8640 10 librados: watch_flush enter 2024-03-29T10:51:17.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.185+0000 7fa977ba8640 10 librados: watch_flush exit 2024-03-29T10:51:17.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.186+0000 7fa977ba8640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T10:51:17.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.186+0000 7fa977ba8640 10 client.4439.objecter successfully canceled tick 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 >> v1:172.21.15.174:6816/2473791532 conn(0x7fa95406e4c0 legacy=0x7fa954070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 10 monclient: shutdown 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 >> v1:172.21.15.64:6790/0 conn(0x7fa9700a9d70 legacy=0x7fa970133120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 shutdown_connections 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 >> 172.21.15.64:0/1325867660 conn(0x7fa970066a00 msgr2=0x7fa9700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 shutdown_connections 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.187+0000 7fa977ba8640 1 -- 172.21.15.64:0/1325867660 wait complete. 2024-03-29T10:51:17.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:17.188+0000 7fa977ba8640 1 librados: shutdown 2024-03-29T10:51:17.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:23.995 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.174+0000 7ff9c3747640 10 monclient: get_monmap_and_config 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.174+0000 7ff9c3747640 10 monclient: build_initial_monmap 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.174+0000 7ff9c3747640 10 monclient: monmap: 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:24.175706+0000 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:24.175706+0000 2024-03-29T10:51:24.157 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.176+0000 7ff9c3747640 1 Processor -- start 2024-03-29T10:51:24.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.176+0000 7ff9c3747640 1 -- start start 2024-03-29T10:51:24.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.176+0000 7ff9c3747640 10 monclient: init 2024-03-29T10:51:24.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.177+0000 7ff9c3747640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.177+0000 7ff9c3747640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 10 monclient(hunting): picked mon.noname-a con 0x7ff9bc06ebd0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 10 monclient(hunting): picked mon.noname-c con 0x7ff9bc06efb0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 10 monclient(hunting): picked mon.noname-b con 0x7ff9bc13ed90 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc070b40 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc070e20 con 0x7ff9bc06efb0 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc071100 con 0x7ff9bc13ed90 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:24.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.178+0000 7ff9c3747640 10 monclient(hunting): authenticate will time out at 1095.477652s 2024-03-29T10:51:24.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.181+0000 7ff9c14e3640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff9bc06ebd0 0x7ff9bc13b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34950/0 (socket says 172.21.15.64:34950) 2024-03-29T10:51:24.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.181+0000 7ff9c14e3640 1 -- 172.21.15.64:0/2234201772 learned_addr learned my addr 172.21.15.64:0/2234201772 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:24.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12336200 0 0) 0x7ff9bc070b40 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a8003520 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893162652 0 0) 0x7ff9bc070e20 con 0x7ff9bc06efb0 2024-03-29T10:51:24.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:24.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:24.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:24.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:51:24.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9bc070b40 con 0x7ff9bc06efb0 2024-03-29T10:51:24.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886866214 0 0) 0x7ff9bc071100 con 0x7ff9bc13ed90 2024-03-29T10:51:24.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:24.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:24.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:24.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:51:24.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9bc070e20 con 0x7ff9bc13ed90 2024-03-29T10:51:24.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1161180878 0 0) 0x7ff9a8003520 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 >> v1:172.21.15.64:6790/0 conn(0x7ff9bc13ed90 legacy=0x7ff9bc149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 >> v1:172.21.15.174:6789/0 conn(0x7ff9bc06efb0 legacy=0x7ff9bc13e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 monclient: found mon.noname-a 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.182+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc13c040 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff9b8001690 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9b3fff640 10 monclient: dump: 2024-03-29T10:51:24.177 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:24.178 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9c3747640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.183+0000 7ff9c3747640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9b80018b0 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9b3fff640 1 -- 172.21.15.64:0/2234201772 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff9b8001c40 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9c3747640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9c3747640 4 set_mon_vals no callback set 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.186+0000 7ff9b3fff640 10 monclient: dump: 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:24.180 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.181 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.181 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:24.181 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.181 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:24.181 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:24.181 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.187+0000 7ff9c3747640 10 monclient: shutdown 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.187+0000 7ff9c3747640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.187+0000 7ff9c3747640 1 -- 172.21.15.64:0/2234201772 >> v1:172.21.15.64:6789/0 conn(0x7ff9bc06ebd0 legacy=0x7ff9bc13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.188+0000 7ff9c3747640 1 -- 172.21.15.64:0/2234201772 shutdown_connections 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.188+0000 7ff9c3747640 1 -- 172.21.15.64:0/2234201772 >> 172.21.15.64:0/2234201772 conn(0x7ff9bc0b4200 msgr2=0x7ff9bc0b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:24.182 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.189+0000 7ff9c3747640 1 -- 172.21.15.64:0/2234201772 shutdown_connections 2024-03-29T10:51:24.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.189+0000 7ff9c3747640 1 -- 172.21.15.64:0/2234201772 wait complete. 2024-03-29T10:51:24.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.190+0000 7ff9c3747640 10 monclient: build_initial_monmap 2024-03-29T10:51:24.188 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.190+0000 7ff9c3747640 10 monclient: monmap: 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 1 librados: starting msgr at 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 1 librados: starting objecter 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 1 Processor -- start 2024-03-29T10:51:24.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 1 -- start start 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 1 librados: setting wanted keys 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 1 librados: calling monclient init 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.191+0000 7ff9c3747640 10 monclient: init 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.192+0000 7ff9c3747640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.192+0000 7ff9c3747640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 10 monclient(hunting): picked mon.noname-c con 0x7ff9bc06ebd0 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 10 monclient(hunting): picked mon.noname-a con 0x7ff9bc06efb0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 10 monclient(hunting): picked mon.noname-b con 0x7ff9bc13ed90 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc140820 con 0x7ff9bc06efb0 2024-03-29T10:51:24.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc0abe70 con 0x7ff9bc13ed90 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc0ac150 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c3747640 10 monclient(hunting): authenticate will time out at 1095.494569s 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.195+0000 7ff9c14e3640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff9bc06ebd0 0x7ff9bc0ada90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56343/0 (socket says 172.21.15.64:56343) 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9c14e3640 1 -- 172.21.15.64:0/959268790 learned_addr learned my addr 172.21.15.64:0/959268790 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 client.?.objecter ms_handle_connect 0x7ff9bc06ebd0 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 client.?.objecter ms_handle_connect 0x7ff9bc06efb0 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855784025 0 0) 0x7ff9bc0ac150 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a0004eb0 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967794750 0 0) 0x7ff9bc140820 con 0x7ff9bc06efb0 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9bc0ac150 con 0x7ff9bc06efb0 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.196+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2579668553 0 0) 0x7ff9a0004eb0 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9bc140820 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff9b8001690 con 0x7ff9bc06ebd0 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:24.193 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3940300414 0 0) 0x7ff9bc0ac150 con 0x7ff9bc06efb0 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9a0004eb0 con 0x7ff9bc06efb0 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff9ac001930 con 0x7ff9bc06efb0 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:24.194 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:24.195 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194841493 0 0) 0x7ff9a0004eb0 con 0x7ff9bc06efb0 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 >> v1:172.21.15.64:6790/0 conn(0x7ff9bc06ebd0 legacy=0x7ff9bc0ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.197+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 >> v1:172.21.15.174:6789/0 conn(0x7ff9bc13ed90 legacy=0x7ff9bc0aa1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 1 monclient: found mon.a 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc0ae1a0 con 0x7ff9bc06efb0 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:24.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9c3747640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9c3747640 10 monclient: _renew_subs 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9c3747640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.198+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9bc1df360 con 0x7ff9bc06efb0 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9c3747640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9c3747640 10 monclient: _renew_subs 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9c3747640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9bc0aa8e0 con 0x7ff9bc06efb0 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9c3747640 1 librados: init done 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9ac003610 con 0x7ff9bc06efb0 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ff9ac003a20 con 0x7ff9bc06efb0 2024-03-29T10:51:24.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b1ffb640 10 monclient: dump: 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:24.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.199+0000 7ff9b37fe640 4 set_mon_vals no callback set 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.202+0000 7ff9c3747640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.202+0000 7ff9c3747640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.202+0000 7ff9c3747640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.202+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c005080 con 0x7ff9bc06efb0 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.202+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7ff9ac003b80 con 0x7ff9bc06efb0 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7ff9ac083840 con 0x7ff9bc06efb0 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9b1ffb640 10 client.4416.objecter ms_dispatch 0x7ff9bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9b1ffb640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9b1ffb640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9b1ffb640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T10:51:24.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9c14e3640 10 monclient: get_auth_request con 0x7ff9a006dec0 auth_method 0 2024-03-29T10:51:24.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.203+0000 7ff9b1ffb640 10 client.4416.objecter ms_handle_connect 0x7ff9a006dec0 2024-03-29T10:51:24.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.213+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9ac04f120 con 0x7ff9bc06efb0 2024-03-29T10:51:24.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.213+0000 7ff9b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:24.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.213+0000 7ff9b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:24.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.629+0000 7ff9c3747640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:24.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.629+0000 7ff9c3747640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:24.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.629+0000 7ff9c3747640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:24.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.629+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c005330 con 0x7ff9bc06efb0 2024-03-29T10:51:24.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.630+0000 7ff9b1ffb640 1 -- 172.21.15.64:0/959268790 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7ff9ac051050 con 0x7ff9bc06efb0 2024-03-29T10:51:24.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.630+0000 7ff9b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:24.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.630+0000 7ff9b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:24.612 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:24.613 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:24.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 10 librados: watch_flush enter 2024-03-29T10:51:24.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 10 librados: watch_flush exit 2024-03-29T10:51:24.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T10:51:24.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 10 client.4416.objecter successfully canceled tick 2024-03-29T10:51:24.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 >> v1:172.21.15.174:6816/2473791532 conn(0x7ff9a006dec0 legacy=0x7ff9a0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:24.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 10 monclient: shutdown 2024-03-29T10:51:24.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:24.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.635+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 >> v1:172.21.15.64:6789/0 conn(0x7ff9bc06efb0 legacy=0x7ff9bc1dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:24.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.636+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 shutdown_connections 2024-03-29T10:51:24.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.636+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 >> 172.21.15.64:0/959268790 conn(0x7ff9bc0b4200 msgr2=0x7ff9bc13b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:24.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.636+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 shutdown_connections 2024-03-29T10:51:24.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.636+0000 7ff9c3747640 1 -- 172.21.15.64:0/959268790 wait complete. 2024-03-29T10:51:24.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:24.636+0000 7ff9c3747640 1 librados: shutdown 2024-03-29T10:51:24.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:31.443 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:31.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.609+0000 7f1d94396640 10 monclient: get_monmap_and_config 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.609+0000 7f1d94396640 10 monclient: build_initial_monmap 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.609+0000 7f1d94396640 10 monclient: monmap: 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:31.610016+0000 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:31.610016+0000 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:31.591 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:31.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.610+0000 7f1d94396640 1 Processor -- start 2024-03-29T10:51:31.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.610+0000 7f1d94396640 1 -- start start 2024-03-29T10:51:31.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.610+0000 7f1d94396640 10 monclient: init 2024-03-29T10:51:31.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.611+0000 7f1d94396640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:31.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.611+0000 7f1d94396640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.614+0000 7f1d94396640 10 monclient(hunting): picked mon.noname-c con 0x7f1d8c0b0770 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.614+0000 7f1d94396640 10 monclient(hunting): picked mon.noname-a con 0x7f1d8c14a700 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.614+0000 7f1d94396640 10 monclient(hunting): picked mon.noname-b con 0x7f1d8c0ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.614+0000 7f1d94396640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c0aba70 con 0x7f1d8c14a700 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.615+0000 7f1d94396640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c0aecb0 con 0x7f1d8c0b0770 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.615+0000 7f1d92933640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1d8c0ac5a0 0x7f1d8c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51862/0 (socket says 172.21.15.64:51862) 2024-03-29T10:51:31.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.615+0000 7f1d92933640 1 -- 172.21.15.64:0/1000317471 learned_addr learned my addr 172.21.15.64:0/1000317471 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:31.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.615+0000 7f1d94396640 1 -- 172.21.15.64:0/1000317471 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c0af150 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.615+0000 7f1d94396640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:31.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.615+0000 7f1d94396640 10 monclient(hunting): authenticate will time out at 1102.914503s 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089173470 0 0) 0x7f1d8c0aecb0 con 0x7f1d8c0b0770 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): _negotiate 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d78003520 con 0x7f1d8c0b0770 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820293701 0 0) 0x7f1d8c0af150 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): _negotiate 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.618+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c0aecb0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709956913 0 0) 0x7f1d8c0aba70 con 0x7f1d8c14a700 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient(hunting): _negotiate 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c0af150 con 0x7f1d8c14a700 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 955547004 0 0) 0x7f1d8c0aecb0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 >> v1:172.21.15.174:6789/0 conn(0x7f1d8c0b0770 legacy=0x7f1d8c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 >> v1:172.21.15.64:6789/0 conn(0x7f1d8c14a700 legacy=0x7f1d8c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: _finish_hunting 0 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 monclient: found mon.noname-b 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c0ac9d0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: _finish_auth 0 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: _check_auth_tickets 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1d7c0015f0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:31.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: dump: 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:31.605 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d7c001750 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 1 -- 172.21.15.64:0/1000317471 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1d7c001b60 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.619+0000 7f1d91130640 10 monclient: dump: 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:31.606 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:31.607 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:31.607 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:31.607 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.621+0000 7f1d94396640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.621+0000 7f1d94396640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.621+0000 7f1d94396640 4 set_mon_vals no callback set 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.622+0000 7f1d94396640 10 monclient: shutdown 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.622+0000 7f1d94396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.622+0000 7f1d94396640 1 -- 172.21.15.64:0/1000317471 >> v1:172.21.15.64:6790/0 conn(0x7f1d8c0ac5a0 legacy=0x7f1d8c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.623+0000 7f1d94396640 1 -- 172.21.15.64:0/1000317471 shutdown_connections 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.623+0000 7f1d94396640 1 -- 172.21.15.64:0/1000317471 >> 172.21.15.64:0/1000317471 conn(0x7f1d8c06b660 msgr2=0x7f1d8c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.623+0000 7f1d94396640 1 -- 172.21.15.64:0/1000317471 shutdown_connections 2024-03-29T10:51:31.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.623+0000 7f1d94396640 1 -- 172.21.15.64:0/1000317471 wait complete. 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.624+0000 7f1d94396640 10 monclient: build_initial_monmap 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.624+0000 7f1d94396640 10 monclient: monmap: 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:31.610 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:31.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.624+0000 7f1d94396640 1 librados: starting msgr at 2024-03-29T10:51:31.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.624+0000 7f1d94396640 1 librados: starting objecter 2024-03-29T10:51:31.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.624+0000 7f1d94396640 1 Processor -- start 2024-03-29T10:51:31.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.627+0000 7f1d94396640 1 -- start start 2024-03-29T10:51:31.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.627+0000 7f1d94396640 1 librados: setting wanted keys 2024-03-29T10:51:31.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.627+0000 7f1d94396640 1 librados: calling monclient init 2024-03-29T10:51:31.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.627+0000 7f1d94396640 10 monclient: init 2024-03-29T10:51:31.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.628+0000 7f1d94396640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.628+0000 7f1d94396640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.630+0000 7f1d94396640 10 monclient(hunting): picked mon.noname-b con 0x7f1d8c0ac5a0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d94396640 10 monclient(hunting): picked mon.noname-a con 0x7f1d8c0b0770 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d94396640 10 monclient(hunting): picked mon.noname-c con 0x7f1d8c14a700 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d92132640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f1d8c0ac5a0 0x7f1d8c126e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:48421/0 (socket says 172.21.15.64:48421) 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d92132640 1 -- 172.21.15.64:0/2364500564 learned_addr learned my addr 172.21.15.64:0/2364500564 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c128210 con 0x7f1d8c0b0770 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c1284f0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d82ffd640 10 client.?.objecter ms_handle_connect 0x7f1d8c0ac5a0 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.632+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c1239d0 con 0x7f1d8c14a700 2024-03-29T10:51:31.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d94396640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d94396640 10 monclient(hunting): authenticate will time out at 1102.931954s 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913685599 0 0) 0x7f1d8c1284f0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d82ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d82ffd640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.633+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d70004eb0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 client.?.objecter ms_handle_connect 0x7f1d8c0b0770 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 863403817 0 0) 0x7f1d70004eb0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d70005300 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1d84001cb0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.625 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558221629 0 0) 0x7f1d8c128210 con 0x7f1d8c0b0770 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:31.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.634+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d70004eb0 con 0x7f1d8c0b0770 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 10 client.?.objecter ms_handle_connect 0x7f1d8c14a700 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3246368097 0 0) 0x7f1d70005300 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 >> v1:172.21.15.64:6790/0 conn(0x7f1d8c14a700 legacy=0x7f1d8c1451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 >> v1:172.21.15.64:6789/0 conn(0x7f1d8c0b0770 legacy=0x7f1d8c1df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 1 monclient: found mon.b 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c123b60 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d94396640 5 monclient: authenticate success, global_id 4435 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.635+0000 7f1d94396640 10 monclient: _renew_subs 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c1df740 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d84003990 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1d84003d60 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d82ffd640 10 monclient: dump: 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:31.628 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 10 monclient: _renew_subs 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c1df9f0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.636+0000 7f1d94396640 1 librados: init done 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.637+0000 7f1d9092f640 4 set_mon_vals no callback set 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.638+0000 7f1d94396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.638+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f1d84003ec0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.638+0000 7f1d94396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.638+0000 7f1d94396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.638+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d74005080 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d91931640 10 monclient: get_auth_request con 0x7f1d7006e000 auth_method 0 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f1d84054b90 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d82ffd640 10 client.4435.objecter ms_dispatch 0x7f1d8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d82ffd640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d82ffd640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d82ffd640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T10:51:31.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.639+0000 7f1d82ffd640 10 client.4435.objecter ms_handle_connect 0x7f1d7006e000 2024-03-29T10:51:31.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.649+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d84050e30 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.649+0000 7f1d82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:31.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.649+0000 7f1d82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:31.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.984+0000 7f1d94396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:31.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.984+0000 7f1d94396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:31.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.984+0000 7f1d94396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:31.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.984+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d74005330 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.985+0000 7f1d82ffd640 1 -- 172.21.15.64:0/2364500564 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f1d84050fd0 con 0x7f1d8c0ac5a0 2024-03-29T10:51:31.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.985+0000 7f1d82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:31.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.985+0000 7f1d82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:31.967 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:31.967 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.988+0000 7f1d94396640 10 librados: watch_flush enter 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.988+0000 7f1d94396640 10 librados: watch_flush exit 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.988+0000 7f1d94396640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.988+0000 7f1d94396640 10 client.4435.objecter successfully canceled tick 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 >> v1:172.21.15.174:6816/2473791532 conn(0x7f1d7006e000 legacy=0x7f1d700704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 10 monclient: shutdown 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 >> v1:172.21.15.174:6789/0 conn(0x7f1d8c0ac5a0 legacy=0x7f1d8c126e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 shutdown_connections 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 >> 172.21.15.64:0/2364500564 conn(0x7f1d8c06b660 msgr2=0x7f1d8c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:31.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 shutdown_connections 2024-03-29T10:51:31.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 -- 172.21.15.64:0/2364500564 wait complete. 2024-03-29T10:51:31.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:31.989+0000 7f1d94396640 1 librados: shutdown 2024-03-29T10:51:31.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:38.795 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:38.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.988+0000 7ffb1e596640 10 monclient: get_monmap_and_config 2024-03-29T10:51:38.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.988+0000 7ffb1e596640 10 monclient: build_initial_monmap 2024-03-29T10:51:38.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.988+0000 7ffb1e596640 10 monclient: monmap: 2024-03-29T10:51:38.970 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:38.989776+0000 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:38.989776+0000 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:38.971 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:38.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.990+0000 7ffb1e596640 1 Processor -- start 2024-03-29T10:51:38.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.990+0000 7ffb1e596640 1 -- start start 2024-03-29T10:51:38.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.990+0000 7ffb1e596640 10 monclient: init 2024-03-29T10:51:38.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:38.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 10 monclient(hunting): picked mon.noname-b con 0x7ffb181457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1813f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 10 monclient(hunting): picked mon.noname-c con 0x7ffb180a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180aaef0 con 0x7ffb1813f650 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.991+0000 7ffb1e596640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18145bd0 con 0x7ffb180a9d70 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.992+0000 7ffb1e596640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18145eb0 con 0x7ffb181457f0 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.992+0000 7ffb1e596640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.992+0000 7ffb1e596640 10 monclient(hunting): authenticate will time out at 1110.290957s 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.992+0000 7ffb17fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ffb181457f0 0x7ffb1813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56664/0 (socket says 172.21.15.64:56664) 2024-03-29T10:51:38.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.992+0000 7ffb17fff640 1 -- 172.21.15.64:0/467351058 learned_addr learned my addr 172.21.15.64:0/467351058 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878492950 0 0) 0x7ffb18145eb0 con 0x7ffb181457f0 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb00003520 con 0x7ffb181457f0 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97293408 0 0) 0x7ffb18145bd0 con 0x7ffb180a9d70 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:51:38.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18145eb0 con 0x7ffb180a9d70 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522190107 0 0) 0x7ffb180aaef0 con 0x7ffb1813f650 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.993+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18145bd0 con 0x7ffb1813f650 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126192347 0 0) 0x7ffb18145eb0 con 0x7ffb180a9d70 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 >> v1:172.21.15.64:6790/0 conn(0x7ffb181457f0 legacy=0x7ffb1813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 >> v1:172.21.15.64:6789/0 conn(0x7ffb1813f650 legacy=0x7ffb180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 1 monclient: found mon.noname-c 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:51:38.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18146340 con 0x7ffb180a9d70 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb1e596640 5 monclient: authenticate success, global_id 4438 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ffb08001670 con 0x7ffb180a9d70 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb16ffd640 10 monclient: dump: 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:38.978 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:38.979 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:38.979 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:38.979 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:38.979 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:38.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.994+0000 7ffb1e596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb080017d0 con 0x7ffb180a9d70 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb16ffd640 1 -- 172.21.15.64:0/467351058 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ffb08001be0 con 0x7ffb180a9d70 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb16ffd640 10 monclient: dump: 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.981 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb1e596640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.995+0000 7ffb1e596640 4 set_mon_vals no callback set 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.996+0000 7ffb1e596640 10 monclient: shutdown 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.996+0000 7ffb1e596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.996+0000 7ffb1e596640 1 -- 172.21.15.64:0/467351058 >> v1:172.21.15.174:6789/0 conn(0x7ffb180a9d70 legacy=0x7ffb1814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.996+0000 7ffb1e596640 1 -- 172.21.15.64:0/467351058 shutdown_connections 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.996+0000 7ffb1e596640 1 -- 172.21.15.64:0/467351058 >> 172.21.15.64:0/467351058 conn(0x7ffb18066a00 msgr2=0x7ffb18066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.997+0000 7ffb1e596640 1 -- 172.21.15.64:0/467351058 shutdown_connections 2024-03-29T10:51:38.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.997+0000 7ffb1e596640 1 -- 172.21.15.64:0/467351058 wait complete. 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.998+0000 7ffb1e596640 10 monclient: build_initial_monmap 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.998+0000 7ffb1e596640 10 monclient: monmap: 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 1 librados: starting msgr at 2024-03-29T10:51:38.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 1 librados: starting objecter 2024-03-29T10:51:38.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 1 Processor -- start 2024-03-29T10:51:38.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 1 -- start start 2024-03-29T10:51:38.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 1 librados: setting wanted keys 2024-03-29T10:51:38.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 1 librados: calling monclient init 2024-03-29T10:51:38.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:38.999+0000 7ffb1e596640 10 monclient: init 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient(hunting): picked mon.noname-b con 0x7ffb180a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1813f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient(hunting): picked mon.noname-a con 0x7ffb181457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb17fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ffb180a9d70 0x7ffb181406a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:54174/0 (socket says 172.21.15.64:54174) 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1813a190 con 0x7ffb181457f0 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb17fff640 1 -- 172.21.15.64:0/434277972 learned_addr learned my addr 172.21.15.64:0/434277972 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18140210 con 0x7ffb180a9d70 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18141ab0 con 0x7ffb1813f650 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb1e596640 10 monclient(hunting): authenticate will time out at 1110.299687s 2024-03-29T10:51:38.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb14ff9640 10 client.?.objecter ms_handle_connect 0x7ffb180a9d70 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.000+0000 7ffb14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831593368 0 0) 0x7ffb18140210 con 0x7ffb180a9d70 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04004eb0 con 0x7ffb180a9d70 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 10 client.?.objecter ms_handle_connect 0x7ffb1813f650 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.001+0000 7ffb14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 client.?.objecter ms_handle_connect 0x7ffb181457f0 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714956156 0 0) 0x7ffb18141ab0 con 0x7ffb1813f650 2024-03-29T10:51:38.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18140210 con 0x7ffb1813f650 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871030903 0 0) 0x7ffb04004eb0 con 0x7ffb180a9d70 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb18141ab0 con 0x7ffb180a9d70 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ffb10001c70 con 0x7ffb180a9d70 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.002+0000 7ffb14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.991 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1368749578 0 0) 0x7ffb18141ab0 con 0x7ffb180a9d70 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 >> v1:172.21.15.64:6790/0 conn(0x7ffb1813f650 legacy=0x7ffb181d70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 >> v1:172.21.15.64:6789/0 conn(0x7ffb181457f0 legacy=0x7ffb181d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 1 monclient: found mon.b 2024-03-29T10:51:38.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18141d90 con 0x7ffb180a9d70 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.003+0000 7ffb14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 10 monclient: _renew_subs 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb181d7ed0 con 0x7ffb180a9d70 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 10 monclient: _renew_subs 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1813b450 con 0x7ffb180a9d70 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb1e596640 1 librados: init done 2024-03-29T10:51:38.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb100020e0 con 0x7ffb180a9d70 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb14ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.004+0000 7ffb15ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.005+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7ffb10004dc0 con 0x7ffb180a9d70 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.005+0000 7ffb14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.006+0000 7ffb14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.006+0000 7ffb14ff9640 10 monclient: dump: 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:38.994 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:38.995 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:38.995 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.011+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7ffb10004f20 con 0x7ffb180a9d70 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.011+0000 7ffb1e596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.011+0000 7ffb1e596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.011+0000 7ffb1e596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.011+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18075f10 con 0x7ffb180a9d70 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.012+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7ffb10084e20 con 0x7ffb180a9d70 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.012+0000 7ffb14ff9640 10 client.4441.objecter ms_dispatch 0x7ffb18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.012+0000 7ffb14ff9640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:38.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.012+0000 7ffb14ff9640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:38.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.012+0000 7ffb14ff9640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T10:51:38.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.012+0000 7ffb177fe640 10 monclient: get_auth_request con 0x7ffb0406dd10 auth_method 0 2024-03-29T10:51:38.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.013+0000 7ffb14ff9640 10 client.4441.objecter ms_handle_connect 0x7ffb0406dd10 2024-03-29T10:51:39.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.019+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb10050700 con 0x7ffb180a9d70 2024-03-29T10:51:39.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.019+0000 7ffb14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:39.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.019+0000 7ffb14ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:39.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.385+0000 7ffb1e596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:39.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.385+0000 7ffb1e596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:39.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.385+0000 7ffb1e596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:51:39.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.385+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb1813b880 con 0x7ffb180a9d70 2024-03-29T10:51:39.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.386+0000 7ffb14ff9640 1 -- 172.21.15.64:0/434277972 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7ffb10052630 con 0x7ffb180a9d70 2024-03-29T10:51:39.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.386+0000 7ffb14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:39.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.386+0000 7ffb14ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:39.368 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:39.368 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:39.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 10 librados: watch_flush enter 2024-03-29T10:51:39.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 10 librados: watch_flush exit 2024-03-29T10:51:39.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T10:51:39.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 10 client.4441.objecter successfully canceled tick 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 >> v1:172.21.15.174:6816/2473791532 conn(0x7ffb0406dd10 legacy=0x7ffb040701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 10 monclient: shutdown 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 >> v1:172.21.15.174:6789/0 conn(0x7ffb180a9d70 legacy=0x7ffb181406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 shutdown_connections 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 >> 172.21.15.64:0/434277972 conn(0x7ffb18066a00 msgr2=0x7ffb180b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.390+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 shutdown_connections 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.391+0000 7ffb1e596640 1 -- 172.21.15.64:0/434277972 wait complete. 2024-03-29T10:51:39.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:39.391+0000 7ffb1e596640 1 librados: shutdown 2024-03-29T10:51:39.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:46.195 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:46.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.343+0000 7f791db6c640 10 monclient: get_monmap_and_config 2024-03-29T10:51:46.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.343+0000 7f791db6c640 10 monclient: build_initial_monmap 2024-03-29T10:51:46.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.343+0000 7f791db6c640 10 monclient: monmap: 2024-03-29T10:51:46.325 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:46.325 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:46.325 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:46.344547+0000 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:46.344547+0000 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:46.326 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.345+0000 7f791db6c640 1 Processor -- start 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.345+0000 7f791db6c640 1 -- start start 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.345+0000 7f791db6c640 10 monclient: init 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient(hunting): picked mon.noname-a con 0x7f79181457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient(hunting): picked mon.noname-b con 0x7f791813f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient(hunting): picked mon.noname-c con 0x7f79180a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:46.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79180aaef0 con 0x7f79181457f0 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918145bd0 con 0x7f79180a9d70 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918145eb0 con 0x7f791813f650 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f791db6c640 10 monclient(hunting): authenticate will time out at 1117.645158s 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f7916ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f791813f650 0x7f79180b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46562/0 (socket says 172.21.15.64:46562) 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f7916ffd640 1 -- 172.21.15.64:0/518705030 learned_addr learned my addr 172.21.15.64:0/518705030 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498066591 0 0) 0x7f7918145eb0 con 0x7f791813f650 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.346+0000 7f79167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:51:46.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7900003520 con 0x7f791813f650 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983164738 0 0) 0x7f7918145bd0 con 0x7f79180a9d70 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7918145eb0 con 0x7f79180a9d70 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3821217911 0 0) 0x7f7900003520 con 0x7f791813f650 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 >> v1:172.21.15.174:6789/0 conn(0x7f79180a9d70 legacy=0x7f791814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:46.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.347+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 >> v1:172.21.15.64:6789/0 conn(0x7f79181457f0 legacy=0x7f791813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:46.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:46.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 1 monclient: found mon.noname-b 2024-03-29T10:51:46.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918146340 con 0x7f791813f650 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f790c001690 con 0x7f791813f650 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f79167fc640 10 monclient: dump: 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:46.334 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f791db6c640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.348+0000 7f791db6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c0017f0 con 0x7f791813f650 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f79167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:46.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f79167fc640 1 -- 172.21.15.64:0/518705030 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f790c001c00 con 0x7f791813f650 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f79167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f79167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f79167fc640 10 monclient: dump: 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:46.336 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f791db6c640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.349+0000 7f791db6c640 4 set_mon_vals no callback set 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.350+0000 7f791db6c640 10 monclient: shutdown 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.350+0000 7f791db6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.350+0000 7f791db6c640 1 -- 172.21.15.64:0/518705030 >> v1:172.21.15.64:6790/0 conn(0x7f791813f650 legacy=0x7f79180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.350+0000 7f791db6c640 1 -- 172.21.15.64:0/518705030 shutdown_connections 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.350+0000 7f791db6c640 1 -- 172.21.15.64:0/518705030 >> 172.21.15.64:0/518705030 conn(0x7f7918066a00 msgr2=0x7f7918066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.351+0000 7f791db6c640 1 -- 172.21.15.64:0/518705030 shutdown_connections 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.351+0000 7f791db6c640 1 -- 172.21.15.64:0/518705030 wait complete. 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.351+0000 7f791db6c640 10 monclient: build_initial_monmap 2024-03-29T10:51:46.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.351+0000 7f791db6c640 10 monclient: monmap: 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.352+0000 7f791db6c640 1 librados: starting msgr at 2024-03-29T10:51:46.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.352+0000 7f791db6c640 1 librados: starting objecter 2024-03-29T10:51:46.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.352+0000 7f791db6c640 1 Processor -- start 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.353+0000 7f791db6c640 1 -- start start 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.353+0000 7f791db6c640 1 librados: setting wanted keys 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.353+0000 7f791db6c640 1 librados: calling monclient init 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.353+0000 7f791db6c640 10 monclient: init 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:46.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 10 monclient(hunting): picked mon.noname-a con 0x7f79180a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 10 monclient(hunting): picked mon.noname-c con 0x7f791813f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 10 monclient(hunting): picked mon.noname-b con 0x7f79181457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791813bb50 con 0x7f79180a9d70 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f7917fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f79181457f0 0x7f79180b0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:33534/0 (socket says 172.21.15.64:33534) 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f7917fff640 1 -- 172.21.15.64:0/2207821769 learned_addr learned my addr 172.21.15.64:0/2207821769 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.354+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79181d7e00 con 0x7f79181457f0 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 client.?.objecter ms_handle_connect 0x7f79181457f0 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 client.?.objecter ms_handle_connect 0x7f791813f650 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79180b0d60 con 0x7f791813f650 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760743541 0 0) 0x7f79181d7e00 con 0x7f79181457f0 2024-03-29T10:51:46.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f791db6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f791db6c640 10 monclient(hunting): authenticate will time out at 1117.654364s 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7904004eb0 con 0x7f79181457f0 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288275768 0 0) 0x7f79180b0d60 con 0x7f791813f650 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:46.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79181d7e00 con 0x7f791813f650 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 379654274 0 0) 0x7f79181d7e00 con 0x7f791813f650 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79180b0d60 con 0x7f791813f650 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f790c001bc0 con 0x7f791813f650 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): dump: 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.350 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.355+0000 7f78fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1103753944 0 0) 0x7f79180b0d60 con 0x7f791813f650 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 >> v1:172.21.15.174:6789/0 conn(0x7f79181457f0 legacy=0x7f79180b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 >> v1:172.21.15.64:6789/0 conn(0x7f79180a9d70 legacy=0x7f7918142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:46.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 monclient: found mon.c 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180b0ef0 con 0x7f791813f650 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: _finish_auth 0 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c004a90 con 0x7f791813f650 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f790c004da0 con 0x7f791813f650 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:46.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f78fffff640 10 monclient: dump: 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:46.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.356+0000 7f7915ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 5 monclient: authenticate success, global_id 4460 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 10 monclient: _renew_subs 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791813fda0 con 0x7f791813f650 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 10 monclient: _renew_subs 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79181402d0 con 0x7f791813f650 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.357+0000 7f791db6c640 1 librados: init done 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.359+0000 7f791db6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:46.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.359+0000 7f791db6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.359+0000 7f791db6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.359+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79181d7f90 con 0x7f791813f650 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.359+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f790c01ad70 con 0x7f791813f650 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.360+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f790c084b60 con 0x7f791813f650 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.360+0000 7f78fffff640 10 client.4460.objecter ms_dispatch 0x7f7918000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.360+0000 7f78fffff640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.360+0000 7f78fffff640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.360+0000 7f78fffff640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.360+0000 7f79177fe640 10 monclient: get_auth_request con 0x7f790406dd30 auth_method 0 2024-03-29T10:51:46.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.361+0000 7f78fffff640 10 client.4460.objecter ms_handle_connect 0x7f790406dd30 2024-03-29T10:51:46.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.367+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f790c050440 con 0x7f791813f650 2024-03-29T10:51:46.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.367+0000 7f78fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:46.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.367+0000 7f78fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.669+0000 7f791db6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.669+0000 7f791db6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.669+0000 7f791db6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.669+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7918075f10 con 0x7f791813f650 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.670+0000 7f78fffff640 1 -- 172.21.15.64:0/2207821769 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f790c052370 con 0x7f791813f650 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.670+0000 7f78fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.670+0000 7f78fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:46.652 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:46.653 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.673+0000 7f791db6c640 10 librados: watch_flush enter 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 10 librados: watch_flush exit 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 10 client.4460.objecter successfully canceled tick 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 >> v1:172.21.15.174:6816/2473791532 conn(0x7f790406dd30 legacy=0x7f79040701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 10 monclient: shutdown 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 >> v1:172.21.15.64:6790/0 conn(0x7f791813f650 legacy=0x7f79181d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 shutdown_connections 2024-03-29T10:51:46.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 >> 172.21.15.64:0/2207821769 conn(0x7f7918066a00 msgr2=0x7f79180b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:46.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 shutdown_connections 2024-03-29T10:51:46.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 -- 172.21.15.64:0/2207821769 wait complete. 2024-03-29T10:51:46.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:46.674+0000 7f791db6c640 1 librados: shutdown 2024-03-29T10:51:46.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:53.477 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:53.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.646+0000 7f3e04855640 10 monclient: get_monmap_and_config 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.646+0000 7f3e04855640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.647+0000 7f3e04855640 10 monclient: monmap: 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:51:53.647827+0000 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:51:53.647827+0000 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:51:53.629 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:53.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.648+0000 7f3e04855640 1 Processor -- start 2024-03-29T10:51:53.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.648+0000 7f3e04855640 1 -- start start 2024-03-29T10:51:53.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.648+0000 7f3e04855640 10 monclient: init 2024-03-29T10:51:53.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:53.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient(hunting): picked mon.noname-c con 0x7f3e001422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:53.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient(hunting): picked mon.noname-a con 0x7f3e00142690 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:53.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient(hunting): picked mon.noname-b con 0x7f3e000b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:53.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e00142c80 con 0x7f3e00142690 2024-03-29T10:51:53.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e000aae40 con 0x7f3e001422b0 2024-03-29T10:51:53.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0013fbd0 con 0x7f3e000b17d0 2024-03-29T10:51:53.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.649+0000 7f3e04855640 10 monclient(hunting): authenticate will time out at 1124.948024s 2024-03-29T10:51:53.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.651+0000 7f3dfed9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3e000b17d0 0x7f3e000b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47613/0 (socket says 172.21.15.64:47613) 2024-03-29T10:51:53.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.651+0000 7f3dfed9d640 1 -- 172.21.15.64:0/1462964852 learned_addr learned my addr 172.21.15.64:0/1462964852 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:53.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413652486 0 0) 0x7f3e0013fbd0 con 0x7f3e000b17d0 2024-03-29T10:51:53.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de8003520 con 0x7f3e000b17d0 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2214663172 0 0) 0x7f3de8003520 con 0x7f3e000b17d0 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 >> v1:172.21.15.174:6789/0 conn(0x7f3e001422b0 legacy=0x7f3e0013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 >> v1:172.21.15.64:6789/0 conn(0x7f3e00142690 legacy=0x7f3e000b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 monclient: found mon.noname-b 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.652+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00151940 con 0x7f3e000b17d0 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3df8001a10 con 0x7f3e000b17d0 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 10 monclient: dump: 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:53.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3e04855640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3e04855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 1 -- 172.21.15.64:0/1462964852 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3df8001b70 con 0x7f3e000b17d0 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3dfd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3e04855640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.653+0000 7f3e04855640 4 set_mon_vals no callback set 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.654+0000 7f3e04855640 10 monclient: shutdown 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.654+0000 7f3e04855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.654+0000 7f3e04855640 1 -- 172.21.15.64:0/1462964852 >> v1:172.21.15.64:6790/0 conn(0x7f3e000b17d0 legacy=0x7f3e000b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.655+0000 7f3e04855640 1 -- 172.21.15.64:0/1462964852 shutdown_connections 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.655+0000 7f3e04855640 1 -- 172.21.15.64:0/1462964852 >> 172.21.15.64:0/1462964852 conn(0x7f3e00066a00 msgr2=0x7f3e00066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.655+0000 7f3e04855640 1 -- 172.21.15.64:0/1462964852 shutdown_connections 2024-03-29T10:51:53.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.655+0000 7f3e04855640 1 -- 172.21.15.64:0/1462964852 wait complete. 2024-03-29T10:51:53.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.656+0000 7f3e04855640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.656+0000 7f3e04855640 10 monclient: monmap: 2024-03-29T10:51:53.641 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:51:53.641 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:53.641 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 1 librados: starting msgr at 2024-03-29T10:51:53.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 1 librados: starting objecter 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 1 Processor -- start 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 1 -- start start 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 1 librados: setting wanted keys 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 1 librados: calling monclient init 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.657+0000 7f3e04855640 10 monclient: init 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.658+0000 7f3e04855640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.658+0000 7f3e04855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 10 monclient(hunting): picked mon.noname-a con 0x7f3e000b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 10 monclient(hunting): picked mon.noname-b con 0x7f3e001422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 10 monclient(hunting): picked mon.noname-c con 0x7f3e00142690 addr v1:172.21.15.64:6790/0 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e001df5b0 con 0x7f3e000b17d0 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e001df740 con 0x7f3e001422b0 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e00130eb0 con 0x7f3e00142690 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3e04855640 10 monclient(hunting): authenticate will time out at 1124.959536s 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3dfe59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3e000b17d0 0x7f3e00132450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58099/0 (socket says 172.21.15.64:58099) 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3dfe59c640 1 -- 172.21.15.64:0/1813031239 learned_addr learned my addr 172.21.15.64:0/1813031239 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3deeffd640 10 client.?.objecter ms_handle_connect 0x7f3e000b17d0 2024-03-29T10:51:53.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.660+0000 7f3deeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177536865 0 0) 0x7f3e001df5b0 con 0x7f3e000b17d0 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de0004eb0 con 0x7f3e000b17d0 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665840088 0 0) 0x7f3de0004eb0 con 0x7f3e000b17d0 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e001df5b0 con 0x7f3e000b17d0 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3df4001a30 con 0x7f3e000b17d0 2024-03-29T10:51:53.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): dump: 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:53.646 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:53.647 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:53.647 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:53.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.661+0000 7f3deeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:53.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.662+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103479447 0 0) 0x7f3e001df5b0 con 0x7f3e000b17d0 2024-03-29T10:51:53.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.662+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 >> v1:172.21.15.64:6790/0 conn(0x7f3e00142690 legacy=0x7f3e001e63e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 >> v1:172.21.15.174:6789/0 conn(0x7f3e001422b0 legacy=0x7f3e001deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 1 monclient: found mon.a 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00131040 con 0x7f3e000b17d0 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 client.?.objecter ms_handle_connect 0x7f3e001422b0 2024-03-29T10:51:53.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 10 monclient: _renew_subs 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e00131fc0 con 0x7f3e000b17d0 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3df4001d30 con 0x7f3e000b17d0 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f3df4003a60 con 0x7f3e000b17d0 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3deeffd640 10 monclient: dump: 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:53.649 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 10 monclient: _renew_subs 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0013bd80 con 0x7f3e000b17d0 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.664+0000 7f3e04855640 1 librados: init done 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.665+0000 7f3deffff640 4 set_mon_vals no callback set 2024-03-29T10:51:53.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.666+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f3df4003bc0 con 0x7f3e000b17d0 2024-03-29T10:51:53.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.667+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f3df4084b50 con 0x7f3e000b17d0 2024-03-29T10:51:53.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.667+0000 7f3deeffd640 10 client.4431.objecter ms_dispatch 0x7f3e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:53.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.667+0000 7f3deeffd640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:53.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.667+0000 7f3deeffd640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:53.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.667+0000 7f3deeffd640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T10:51:53.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.667+0000 7f3dfdd9b640 10 monclient: get_auth_request con 0x7f3de006db10 auth_method 0 2024-03-29T10:51:53.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.669+0000 7f3deeffd640 10 client.4431.objecter ms_handle_connect 0x7f3de006db10 2024-03-29T10:51:53.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.669+0000 7f3e04855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.669+0000 7f3e04855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.669+0000 7f3e04855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:53.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.669+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de4005080 con 0x7f3e000b17d0 2024-03-29T10:51:53.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.677+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3df40502c0 con 0x7f3e000b17d0 2024-03-29T10:51:53.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.677+0000 7f3deeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:53.677+0000 7f3deeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:53.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.008+0000 7f3e04855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:53.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.008+0000 7f3e04855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:53.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.008+0000 7f3e04855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:51:53.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.009+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3de4005330 con 0x7f3e000b17d0 2024-03-29T10:51:53.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.010+0000 7f3deeffd640 1 -- 172.21.15.64:0/1813031239 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f3df4084200 con 0x7f3e000b17d0 2024-03-29T10:51:53.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.011+0000 7f3deeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:53.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.011+0000 7f3deeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:53.993 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:51:53.993 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:53.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 10 librados: watch_flush enter 2024-03-29T10:51:53.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 10 librados: watch_flush exit 2024-03-29T10:51:53.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T10:51:53.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 10 client.4431.objecter successfully canceled tick 2024-03-29T10:51:53.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 >> v1:172.21.15.174:6816/2473791532 conn(0x7f3de006db10 legacy=0x7f3de006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 10 monclient: shutdown 2024-03-29T10:51:53.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.015+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 >> v1:172.21.15.64:6789/0 conn(0x7f3e000b17d0 legacy=0x7f3e00132450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.016+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 shutdown_connections 2024-03-29T10:51:53.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.016+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 >> 172.21.15.64:0/1813031239 conn(0x7f3e00066a00 msgr2=0x7f3e000b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.016+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 shutdown_connections 2024-03-29T10:51:53.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.017+0000 7f3e04855640 1 -- 172.21.15.64:0/1813031239 wait complete. 2024-03-29T10:51:53.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:51:54.017+0000 7f3e04855640 1 librados: shutdown 2024-03-29T10:51:54.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:00.824 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:00.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.988+0000 7fdf3f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.988+0000 7fdf3f59e640 10 monclient: build_initial_monmap 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.988+0000 7fdf3f59e640 10 monclient: monmap: 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:00.989148+0000 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:00.989148+0000 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:00.970 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:00.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.989+0000 7fdf3f59e640 1 Processor -- start 2024-03-29T10:52:00.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.989+0000 7fdf3f59e640 1 -- start start 2024-03-29T10:52:00.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.989+0000 7fdf3f59e640 10 monclient: init 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdf38006890 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdf38006c70 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf3802b4c0 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3802c830 con 0x7fdf38006c70 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3802afa0 con 0x7fdf38006890 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3802aba0 con 0x7fdf3802b4c0 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:00.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.990+0000 7fdf3f59e640 10 monclient(hunting): authenticate will time out at 1132.289222s 2024-03-29T10:52:00.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3e59c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fdf38006890 0x7fdf380c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:36772/0 (socket says 172.21.15.64:36772) 2024-03-29T10:52:00.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3e59c640 1 -- 172.21.15.64:0/3636912151 learned_addr learned my addr 172.21.15.64:0/3636912151 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:00.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108645568 0 0) 0x7fdf3802afa0 con 0x7fdf38006890 2024-03-29T10:52:00.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:00.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:00.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:00.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3d59a640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:52:00.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.992+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf28003440 con 0x7fdf38006890 2024-03-29T10:52:00.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.993+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3579978869 0 0) 0x7fdf28003440 con 0x7fdf38006890 2024-03-29T10:52:00.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.993+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 >> v1:172.21.15.64:6790/0 conn(0x7fdf3802b4c0 legacy=0x7fdf380cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:00.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 >> v1:172.21.15.64:6789/0 conn(0x7fdf38006c70 legacy=0x7fdf380c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:00.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:00.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 1 monclient: found mon.noname-c 2024-03-29T10:52:00.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3802ad30 con 0x7fdf38006890 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: _finish_auth 0 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3f59e640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdf4004f730 con 0x7fdf38006890 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3d59a640 10 monclient: dump: 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:00.977 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.994+0000 7fdf3f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:00.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.997+0000 7fdf3d59a640 1 -- 172.21.15.64:0/3636912151 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf40075f10 con 0x7fdf38006890 2024-03-29T10:52:00.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.997+0000 7fdf3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:00.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.997+0000 7fdf3f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:00.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.997+0000 7fdf3f59e640 4 set_mon_vals no callback set 2024-03-29T10:52:00.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.998+0000 7fdf3f59e640 10 monclient: shutdown 2024-03-29T10:52:00.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.998+0000 7fdf3f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:00.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.998+0000 7fdf3f59e640 1 -- 172.21.15.64:0/3636912151 >> v1:172.21.15.174:6789/0 conn(0x7fdf38006890 legacy=0x7fdf380c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:00.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.998+0000 7fdf3f59e640 1 -- 172.21.15.64:0/3636912151 shutdown_connections 2024-03-29T10:52:00.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.998+0000 7fdf3f59e640 1 -- 172.21.15.64:0/3636912151 >> 172.21.15.64:0/3636912151 conn(0x7fdf380058f0 msgr2=0x7fdf38005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:00.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:00.998+0000 7fdf3f59e640 1 -- 172.21.15.64:0/3636912151 shutdown_connections 2024-03-29T10:52:00.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.001+0000 7fdf3f59e640 1 -- 172.21.15.64:0/3636912151 wait complete. 2024-03-29T10:52:00.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.002+0000 7fdf3f59e640 10 monclient: build_initial_monmap 2024-03-29T10:52:00.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.002+0000 7fdf3f59e640 10 monclient: monmap: 2024-03-29T10:52:00.983 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:00.983 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:00.983 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:00.984 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:00.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.002+0000 7fdf3f59e640 1 librados: starting msgr at 2024-03-29T10:52:00.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.002+0000 7fdf3f59e640 1 librados: starting objecter 2024-03-29T10:52:00.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.003+0000 7fdf3f59e640 1 Processor -- start 2024-03-29T10:52:00.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.003+0000 7fdf3f59e640 1 -- start start 2024-03-29T10:52:00.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.003+0000 7fdf3f59e640 1 librados: setting wanted keys 2024-03-29T10:52:00.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.003+0000 7fdf3f59e640 1 librados: calling monclient init 2024-03-29T10:52:00.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.003+0000 7fdf3f59e640 10 monclient: init 2024-03-29T10:52:00.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.003+0000 7fdf3f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:00.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdf38006890 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf3802b4c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdf380c4470 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3802d0b0 con 0x7fdf380c4470 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3802dd50 con 0x7fdf3802b4c0 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3802dee0 con 0x7fdf38006890 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.004+0000 7fdf3f59e640 10 monclient(hunting): authenticate will time out at 1132.302954s 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.006+0000 7fdf3dd9b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fdf3802b4c0 0x7fdf3815b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:37342/0 (socket says 172.21.15.64:37342) 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.006+0000 7fdf3dd9b640 1 -- 172.21.15.64:0/1254393461 learned_addr learned my addr 172.21.15.64:0/1254393461 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 client.?.objecter ms_handle_connect 0x7fdf3802b4c0 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:00.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 client.?.objecter ms_handle_connect 0x7fdf38006890 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886170100 0 0) 0x7fdf3802dd50 con 0x7fdf3802b4c0 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf20004eb0 con 0x7fdf3802b4c0 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851332326 0 0) 0x7fdf3802dee0 con 0x7fdf38006890 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.007+0000 7fdf2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.008+0000 7fdf2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.008+0000 7fdf2effd640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:52:00.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.008+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3802dd50 con 0x7fdf38006890 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 229984972 0 0) 0x7fdf20004eb0 con 0x7fdf3802b4c0 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf3802dee0 con 0x7fdf3802b4c0 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdf34001920 con 0x7fdf3802b4c0 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): dump: 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:00.996 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148292391 0 0) 0x7fdf3802dd50 con 0x7fdf38006890 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf20004eb0 con 0x7fdf38006890 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdf4004f400 con 0x7fdf38006890 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 10 monclient(hunting): dump: 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:00.997 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 231592288 0 0) 0x7fdf3802dee0 con 0x7fdf3802b4c0 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.009+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 >> v1:172.21.15.64:6790/0 conn(0x7fdf38006890 legacy=0x7fdf3802d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 >> v1:172.21.15.64:6789/0 conn(0x7fdf380c4470 legacy=0x7fdf380bce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:00.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 1 monclient: found mon.b 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf38031970 con 0x7fdf3802b4c0 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 10 monclient: _finish_auth 0 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 5 monclient: authenticate success, global_id 4456 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 10 monclient: _renew_subs 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3815c0e0 con 0x7fdf3802b4c0 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 10 monclient: _renew_subs 2024-03-29T10:52:00.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:01.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf380bd700 con 0x7fdf3802b4c0 2024-03-29T10:52:01.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.010+0000 7fdf3f59e640 1 librados: init done 2024-03-29T10:52:01.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.011+0000 7fdf3f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:01.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.011+0000 7fdf3f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:01.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.012+0000 7fdf3f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:01.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.012+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38031be0 con 0x7fdf3802b4c0 2024-03-29T10:52:01.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf340035f0 con 0x7fdf3802b4c0 2024-03-29T10:52:01.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:01.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fdf340039c0 con 0x7fdf3802b4c0 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf2effd640 10 monclient: dump: 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:01.002 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.013+0000 7fdf3cd99640 4 set_mon_vals no callback set 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.015+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fdf34003b20 con 0x7fdf3802b4c0 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fdf34084a20 con 0x7fdf3802b4c0 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf2effd640 10 client.4456.objecter ms_dispatch 0x7fdf40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf2effd640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf2effd640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf2effd640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf3e59c640 10 monclient: get_auth_request con 0x7fdf2006ddf0 auth_method 0 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.016+0000 7fdf2effd640 10 client.4456.objecter ms_handle_connect 0x7fdf2006ddf0 2024-03-29T10:52:01.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.021+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf34050300 con 0x7fdf3802b4c0 2024-03-29T10:52:01.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.021+0000 7fdf2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:01.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.021+0000 7fdf2effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.350+0000 7fdf3f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.350+0000 7fdf3f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.350+0000 7fdf3f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:01.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.350+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf3815c430 con 0x7fdf3802b4c0 2024-03-29T10:52:01.336 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:01.336 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:01.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.354+0000 7fdf2effd640 1 -- 172.21.15.64:0/1254393461 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fdf34052230 con 0x7fdf3802b4c0 2024-03-29T10:52:01.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.354+0000 7fdf2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:01.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.354+0000 7fdf2effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:01.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.357+0000 7fdf3f59e640 10 librados: watch_flush enter 2024-03-29T10:52:01.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 10 librados: watch_flush exit 2024-03-29T10:52:01.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T10:52:01.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 10 client.4456.objecter successfully canceled tick 2024-03-29T10:52:01.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 >> v1:172.21.15.174:6816/2473791532 conn(0x7fdf2006ddf0 legacy=0x7fdf200702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 10 monclient: shutdown 2024-03-29T10:52:01.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:01.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 >> v1:172.21.15.174:6789/0 conn(0x7fdf3802b4c0 legacy=0x7fdf3815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:01.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 shutdown_connections 2024-03-29T10:52:01.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 >> 172.21.15.64:0/1254393461 conn(0x7fdf380058f0 msgr2=0x7fdf38038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:01.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 shutdown_connections 2024-03-29T10:52:01.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 -- 172.21.15.64:0/1254393461 wait complete. 2024-03-29T10:52:01.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:01.358+0000 7fdf3f59e640 1 librados: shutdown 2024-03-29T10:52:01.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:08.165 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:08.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.349+0000 7f7e52460640 10 monclient: get_monmap_and_config 2024-03-29T10:52:08.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.349+0000 7f7e52460640 10 monclient: build_initial_monmap 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.349+0000 7f7e52460640 10 monclient: monmap: 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:08.350927+0000 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:08.350927+0000 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:08.332 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:08.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.350+0000 7f7e52460640 1 Processor -- start 2024-03-29T10:52:08.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.350+0000 7f7e52460640 1 -- start start 2024-03-29T10:52:08.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.350+0000 7f7e52460640 10 monclient: init 2024-03-29T10:52:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient(hunting): picked mon.noname-c con 0x7f7e4c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient(hunting): picked mon.noname-a con 0x7f7e4c13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient(hunting): picked mon.noname-b con 0x7f7e4c0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c0aaef0 con 0x7f7e4c13f650 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c145bd0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c145eb0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.351+0000 7f7e52460640 10 monclient(hunting): authenticate will time out at 1139.651409s 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.356+0000 7f7e509fd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7e4c0a9d70 0x7f7e4c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:44273/0 (socket says 172.21.15.64:44273) 2024-03-29T10:52:08.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.356+0000 7f7e509fd640 1 -- 172.21.15.64:0/2409823138 learned_addr learned my addr 172.21.15.64:0/2409823138 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.356+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766556188 0 0) 0x7f7e4c145eb0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e38003520 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706901547 0 0) 0x7f7e4c145bd0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:08.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:08.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:08.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:52:08.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.357+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e4c145eb0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2462187979 0 0) 0x7f7e4c145eb0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 >> v1:172.21.15.64:6790/0 conn(0x7f7e4c0a9d70 legacy=0x7f7e4c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 >> v1:172.21.15.64:6789/0 conn(0x7f7e4c13f650 legacy=0x7f7e4c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 monclient: found mon.noname-c 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c146340 con 0x7f7e4c1457f0 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: _finish_auth 0 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:08.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7e340015a0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: dump: 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:08.343 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e340017f0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 1 -- 172.21.15.64:0/2409823138 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7e34001b80 con 0x7f7e4c1457f0 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.358+0000 7f7e4affd640 10 monclient: dump: 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:08.344 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.359+0000 7f7e52460640 5 monclient: authenticate success, global_id 4459 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.359+0000 7f7e52460640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:08.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.359+0000 7f7e52460640 4 set_mon_vals no callback set 2024-03-29T10:52:08.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.362+0000 7f7e52460640 10 monclient: shutdown 2024-03-29T10:52:08.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.362+0000 7f7e52460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:08.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.362+0000 7f7e52460640 1 -- 172.21.15.64:0/2409823138 >> v1:172.21.15.174:6789/0 conn(0x7f7e4c1457f0 legacy=0x7f7e4c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:08.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.364+0000 7f7e52460640 1 -- 172.21.15.64:0/2409823138 shutdown_connections 2024-03-29T10:52:08.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.364+0000 7f7e52460640 1 -- 172.21.15.64:0/2409823138 >> 172.21.15.64:0/2409823138 conn(0x7f7e4c066a00 msgr2=0x7f7e4c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:08.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.364+0000 7f7e52460640 1 -- 172.21.15.64:0/2409823138 shutdown_connections 2024-03-29T10:52:08.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.364+0000 7f7e52460640 1 -- 172.21.15.64:0/2409823138 wait complete. 2024-03-29T10:52:08.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.365+0000 7f7e52460640 10 monclient: build_initial_monmap 2024-03-29T10:52:08.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.365+0000 7f7e52460640 10 monclient: monmap: 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:08.349 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:08.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.365+0000 7f7e52460640 1 librados: starting msgr at 2024-03-29T10:52:08.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.365+0000 7f7e52460640 1 librados: starting objecter 2024-03-29T10:52:08.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.365+0000 7f7e52460640 1 Processor -- start 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 1 -- start start 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 1 librados: setting wanted keys 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 1 librados: calling monclient init 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 10 monclient: init 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 10 monclient(hunting): picked mon.noname-a con 0x7f7e4c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 10 monclient(hunting): picked mon.noname-b con 0x7f7e4c13f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.369+0000 7f7e52460640 10 monclient(hunting): picked mon.noname-c con 0x7f7e4c1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.370+0000 7f7e52460640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c144c70 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.370+0000 7f7e52460640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c1452b0 con 0x7f7e4c13f650 2024-03-29T10:52:08.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.370+0000 7f7e52460640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c143cd0 con 0x7f7e4c1457f0 2024-03-29T10:52:08.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.370+0000 7f7e52460640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:08.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.370+0000 7f7e52460640 10 monclient(hunting): authenticate will time out at 1139.669885s 2024-03-29T10:52:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.374+0000 7f7e4b7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f7e4c13f650 0x7f7e4c1e7460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:40699/0 (socket says 172.21.15.64:40699) 2024-03-29T10:52:08.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.374+0000 7f7e4b7fe640 1 -- 172.21.15.64:0/617307469 learned_addr learned my addr 172.21.15.64:0/617307469 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:08.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.375+0000 7f7e48ff9640 10 client.?.objecter ms_handle_connect 0x7f7e4c0a9d70 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640872592 0 0) 0x7f7e4c144c70 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.376+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c004eb0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3905681285 0 0) 0x7f7e2c004eb0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e4c144c70 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7e34001b00 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:08.364 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 client.?.objecter ms_handle_connect 0x7f7e4c13f650 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1970212624 0 0) 0x7f7e4c144c70 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 >> v1:172.21.15.64:6790/0 conn(0x7f7e4c1457f0 legacy=0x7f7e4c1e7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 >> v1:172.21.15.174:6789/0 conn(0x7f7e4c13f650 legacy=0x7f7e4c1e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 monclient: found mon.a 2024-03-29T10:52:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c143e60 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 10 monclient: _renew_subs 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c1d4fa0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 10 monclient: _renew_subs 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c1da0c0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e52460640 1 librados: init done 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.377+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e340038a0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.378+0000 7f7e48ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.378+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7e34003bb0 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.378+0000 7f7e48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.378+0000 7f7e48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.378+0000 7f7e48ff9640 10 monclient: dump: 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:08.367 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:08.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.378+0000 7f7e4a7fc640 4 set_mon_vals no callback set 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.380+0000 7f7e52460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.381+0000 7f7e52460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.381+0000 7f7e52460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.381+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e30005080 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.382+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f7e34019b80 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.383+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f7e34083960 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.383+0000 7f7e48ff9640 10 client.4434.objecter ms_dispatch 0x7f7e4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.383+0000 7f7e48ff9640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:08.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.383+0000 7f7e48ff9640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:08.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.383+0000 7f7e48ff9640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T10:52:08.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.385+0000 7f7e4b7fe640 10 monclient: get_auth_request con 0x7f7e2c06dc70 auth_method 0 2024-03-29T10:52:08.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.387+0000 7f7e48ff9640 10 client.4434.objecter ms_handle_connect 0x7f7e2c06dc70 2024-03-29T10:52:08.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.396+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e3404f240 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.396+0000 7f7e48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:08.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.396+0000 7f7e48ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:08.705 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:08.705 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.722+0000 7f7e52460640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.722+0000 7f7e52460640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.722+0000 7f7e52460640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.722+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e30005330 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.723+0000 7f7e48ff9640 1 -- 172.21.15.64:0/617307469 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f7e34051170 con 0x7f7e4c0a9d70 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.723+0000 7f7e48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:08.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.723+0000 7f7e48ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:08.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 10 librados: watch_flush enter 2024-03-29T10:52:08.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 10 librados: watch_flush exit 2024-03-29T10:52:08.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T10:52:08.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 10 client.4434.objecter successfully canceled tick 2024-03-29T10:52:08.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 >> v1:172.21.15.174:6816/2473791532 conn(0x7f7e2c06dc70 legacy=0x7f7e2c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:08.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 10 monclient: shutdown 2024-03-29T10:52:08.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:08.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.726+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 >> v1:172.21.15.64:6789/0 conn(0x7f7e4c0a9d70 legacy=0x7f7e4c1d4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:08.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.727+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 shutdown_connections 2024-03-29T10:52:08.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.727+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 >> 172.21.15.64:0/617307469 conn(0x7f7e4c066a00 msgr2=0x7f7e4c1daa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:08.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.729+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 shutdown_connections 2024-03-29T10:52:08.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.729+0000 7f7e52460640 1 -- 172.21.15.64:0/617307469 wait complete. 2024-03-29T10:52:08.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:08.730+0000 7f7e52460640 1 librados: shutdown 2024-03-29T10:52:08.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:15.542 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.707+0000 7f61b3fff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.707+0000 7f61b3fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.707+0000 7f61b3fff640 10 monclient: monmap: 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:15.709484+0000 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:15.709484+0000 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:15.690 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.709+0000 7f61b3fff640 1 Processor -- start 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.709+0000 7f61b3fff640 1 -- start start 2024-03-29T10:52:15.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.709+0000 7f61b3fff640 10 monclient: init 2024-03-29T10:52:15.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:15.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:15.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f61ac006a70 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:15.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f61ac02b100 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:15.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f61ac029c30 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:15.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac02a270 con 0x7f61ac02b100 2024-03-29T10:52:15.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac02c440 con 0x7f61ac006a70 2024-03-29T10:52:15.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac0c9440 con 0x7f61ac029c30 2024-03-29T10:52:15.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:15.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.710+0000 7f61b3fff640 10 monclient(hunting): authenticate will time out at 1147.010362s 2024-03-29T10:52:15.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b27fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f61ac02b100 0x7f61ac0cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60611/0 (socket says 172.21.15.64:60611) 2024-03-29T10:52:15.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b27fc640 1 -- 172.21.15.64:0/2665746495 learned_addr learned my addr 172.21.15.64:0/2665746495 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202015918 0 0) 0x7f61ac02a270 con 0x7f61ac02b100 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b1ffb640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.714+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c003520 con 0x7f61ac02b100 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.715+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273965639 0 0) 0x7f619c003520 con 0x7f61ac02b100 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.715+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 >> v1:172.21.15.64:6790/0 conn(0x7f61ac029c30 legacy=0x7f61ac0d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:15.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.716+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 >> v1:172.21.15.174:6789/0 conn(0x7f61ac006a70 legacy=0x7f61ac0c8d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:15.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:15.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:15.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0c95d0 con 0x7f61ac02b100 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f61a4001690 con 0x7f61ac02b100 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:15.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.717+0000 7f61b1ffb640 10 monclient: dump: 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:15.702 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.718+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61a4001860 con 0x7f61ac02b100 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.718+0000 7f61b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.718+0000 7f61b1ffb640 1 -- 172.21.15.64:0/2665746495 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f61a4001cb0 con 0x7f61ac02b100 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.718+0000 7f61b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.718+0000 7f61b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.718+0000 7f61b1ffb640 10 monclient: dump: 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.703 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.704 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:15.704 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.704 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:15.704 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:15.704 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:15.704 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.719+0000 7f61b3fff640 5 monclient: authenticate success, global_id 4437 2024-03-29T10:52:15.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.719+0000 7f61b3fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:15.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.719+0000 7f61b3fff640 4 set_mon_vals no callback set 2024-03-29T10:52:15.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.720+0000 7f61b3fff640 10 monclient: shutdown 2024-03-29T10:52:15.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.720+0000 7f61b3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.720+0000 7f61b3fff640 1 -- 172.21.15.64:0/2665746495 >> v1:172.21.15.64:6789/0 conn(0x7f61ac02b100 legacy=0x7f61ac0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.720+0000 7f61b3fff640 1 -- 172.21.15.64:0/2665746495 shutdown_connections 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.720+0000 7f61b3fff640 1 -- 172.21.15.64:0/2665746495 >> 172.21.15.64:0/2665746495 conn(0x7f61ac0058f0 msgr2=0x7f61ac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.721+0000 7f61b3fff640 1 -- 172.21.15.64:0/2665746495 shutdown_connections 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.721+0000 7f61b3fff640 1 -- 172.21.15.64:0/2665746495 wait complete. 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.722+0000 7f61b3fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.722+0000 7f61b3fff640 10 monclient: monmap: 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:15.706 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.707 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:15.707 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:15.707 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:15.707 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.722+0000 7f61b3fff640 1 librados: starting msgr at 2024-03-29T10:52:15.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.722+0000 7f61b3fff640 1 librados: starting objecter 2024-03-29T10:52:15.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.723+0000 7f61b3fff640 1 Processor -- start 2024-03-29T10:52:15.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.726+0000 7f61b3fff640 1 -- start start 2024-03-29T10:52:15.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.726+0000 7f61b3fff640 1 librados: setting wanted keys 2024-03-29T10:52:15.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.726+0000 7f61b3fff640 1 librados: calling monclient init 2024-03-29T10:52:15.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.726+0000 7f61b3fff640 10 monclient: init 2024-03-29T10:52:15.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f61ac006a70 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f61ac029c30 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f61ac02d380 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac02eba0 con 0x7f61ac029c30 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac02ee80 con 0x7f61ac006a70 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac169820 con 0x7f61ac02d380 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:15.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.727+0000 7f61b3fff640 10 monclient(hunting): authenticate will time out at 1147.026982s 2024-03-29T10:52:15.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.731+0000 7f61b2ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f61ac006a70 0x7f61ac169d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:58005/0 (socket says 172.21.15.64:58005) 2024-03-29T10:52:15.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.731+0000 7f61b2ffd640 1 -- 172.21.15.64:0/3626460611 learned_addr learned my addr 172.21.15.64:0/3626460611 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 10 client.?.objecter ms_handle_connect 0x7f61ac006a70 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831300274 0 0) 0x7f61ac02ee80 con 0x7f61ac006a70 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.732+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0004eb0 con 0x7f61ac006a70 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 client.?.objecter ms_handle_connect 0x7f61ac02d380 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118344536 0 0) 0x7f61a0004eb0 con 0x7f61ac006a70 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a0005300 con 0x7f61ac006a70 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f61a8001830 con 0x7f61ac006a70 2024-03-29T10:52:15.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:15.722 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274989911 0 0) 0x7f61ac169820 con 0x7f61ac02d380 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:15.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:15.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:15.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:52:15.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0005850 con 0x7f61ac02d380 2024-03-29T10:52:15.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 client.?.objecter ms_handle_connect 0x7f61ac029c30 2024-03-29T10:52:15.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.733+0000 7f619b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:15.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865730259 0 0) 0x7f61ac02eba0 con 0x7f61ac029c30 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0005b90 con 0x7f61ac029c30 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2988261163 0 0) 0x7f61a0005850 con 0x7f61ac02d380 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a00072e0 con 0x7f61ac02d380 2024-03-29T10:52:15.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f61b404fdd0 con 0x7f61ac02d380 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:15.726 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 301511584 0 0) 0x7f61a00072e0 con 0x7f61ac02d380 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 >> v1:172.21.15.174:6789/0 conn(0x7f61ac006a70 legacy=0x7f61ac169d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 >> v1:172.21.15.64:6789/0 conn(0x7f61ac029c30 legacy=0x7f61ac1666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 monclient: found mon.c 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac1694b0 con 0x7f61ac02d380 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:15.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f619b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f61b3fff640 5 monclient: authenticate success, global_id 4472 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f61b3fff640 10 monclient: _renew_subs 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f61b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.734+0000 7f61b3fff640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac166de0 con 0x7f61ac02d380 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.735+0000 7f61b3fff640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.735+0000 7f61b3fff640 10 monclient: _renew_subs 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.735+0000 7f61b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.735+0000 7f61b3fff640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac02cd40 con 0x7f61ac02d380 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.735+0000 7f61b3fff640 1 librados: init done 2024-03-29T10:52:15.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b404f400 con 0x7f61ac02d380 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f619b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f61b404fdd0 con 0x7f61ac02d380 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f619b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f619b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f619b7fe640 10 monclient: dump: 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:15.729 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f61997fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.736+0000 7f61b17fa640 4 set_mon_vals no callback set 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.737+0000 7f61997fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.737+0000 7f61997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.738+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac1699b0 con 0x7f61ac02d380 2024-03-29T10:52:15.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.738+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f61b4075710 con 0x7f61ac02d380 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.739+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f61b40cda90 con 0x7f61ac02d380 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.739+0000 7f619b7fe640 10 client.4472.objecter ms_dispatch 0x7f61b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.739+0000 7f619b7fe640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.739+0000 7f619b7fe640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.739+0000 7f619b7fe640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.743+0000 7f61b2ffd640 10 monclient: get_auth_request con 0x7f61a006eb50 auth_method 0 2024-03-29T10:52:15.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.744+0000 7f619b7fe640 10 client.4472.objecter ms_handle_connect 0x7f61a006eb50 2024-03-29T10:52:15.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.750+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61b40c9d30 con 0x7f61ac02d380 2024-03-29T10:52:15.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.750+0000 7f619b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:15.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:15.750+0000 7f619b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:16.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.095+0000 7f61997fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:16.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.095+0000 7f61997fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:16.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.095+0000 7f61997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:16.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.095+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61ac005e40 con 0x7f61ac02d380 2024-03-29T10:52:16.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.099+0000 7f619b7fe640 1 -- 172.21.15.64:0/3626460611 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f61b40c9ed0 con 0x7f61ac02d380 2024-03-29T10:52:16.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.099+0000 7f619b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:16.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.099+0000 7f619b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:16.083 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:16.083 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:16.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 10 librados: watch_flush enter 2024-03-29T10:52:16.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 10 librados: watch_flush exit 2024-03-29T10:52:16.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T10:52:16.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 10 client.4472.objecter successfully canceled tick 2024-03-29T10:52:16.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 >> v1:172.21.15.174:6816/2473791532 conn(0x7f61a006eb50 legacy=0x7f61a0071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 10 monclient: shutdown 2024-03-29T10:52:16.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:16.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.103+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 >> v1:172.21.15.64:6790/0 conn(0x7f61ac02d380 legacy=0x7f61ac174d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.104+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 shutdown_connections 2024-03-29T10:52:16.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.104+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 >> 172.21.15.64:0/3626460611 conn(0x7f61ac0058f0 msgr2=0x7f61ac0c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:16.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.104+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 shutdown_connections 2024-03-29T10:52:16.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.104+0000 7f61997fa640 1 -- 172.21.15.64:0/3626460611 wait complete. 2024-03-29T10:52:16.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:16.104+0000 7f61997fa640 1 librados: shutdown 2024-03-29T10:52:16.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:22.912 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:23.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.095+0000 7f2477d04640 10 monclient: get_monmap_and_config 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.095+0000 7f2477d04640 10 monclient: build_initial_monmap 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.095+0000 7f2477d04640 10 monclient: monmap: 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:23.096194+0000 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:23.096194+0000 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:23.077 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.096+0000 7f2477d04640 1 Processor -- start 2024-03-29T10:52:23.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.096+0000 7f2477d04640 1 -- start start 2024-03-29T10:52:23.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.096+0000 7f2477d04640 10 monclient: init 2024-03-29T10:52:23.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:23.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:23.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient(hunting): picked mon.noname-b con 0x7f24701457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:23.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient(hunting): picked mon.noname-a con 0x7f247013f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:23.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient(hunting): picked mon.noname-c con 0x7f24700a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24700aaef0 con 0x7f247013f650 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2470145bd0 con 0x7f24700a9d70 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2470145eb0 con 0x7f24701457f0 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2477d04640 10 monclient(hunting): authenticate will time out at 1154.396486s 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2475aa0640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f24701457f0 0x7f247013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45084/0 (socket says 172.21.15.64:45084) 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.097+0000 7f2475aa0640 1 -- 172.21.15.64:0/1111597806 learned_addr learned my addr 172.21.15.64:0/1111597806 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120355974 0 0) 0x7f2470145eb0 con 0x7f24701457f0 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 10 monclient(hunting): _negotiate 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f245c003520 con 0x7f24701457f0 2024-03-29T10:52:23.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888413234 0 0) 0x7f245c003520 con 0x7f24701457f0 2024-03-29T10:52:23.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 >> v1:172.21.15.174:6789/0 conn(0x7f24700a9d70 legacy=0x7f247014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:23.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.098+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 >> v1:172.21.15.64:6789/0 conn(0x7f247013f650 legacy=0x7f24700b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:23.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: _finish_hunting 0 2024-03-29T10:52:23.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 1 monclient: found mon.noname-b 2024-03-29T10:52:23.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:23.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:52:23.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470146340 con 0x7f24701457f0 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: _finish_auth 0 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: _check_auth_tickets 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2477d04640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2477d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f246c001a10 con 0x7f24701457f0 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.101+0000 7f2474a9e640 10 monclient: dump: 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:23.085 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2477d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f246c001ca0 con 0x7f24701457f0 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2474a9e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2477d04640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2477d04640 4 set_mon_vals no callback set 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2474a9e640 1 -- 172.21.15.64:0/1111597806 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f246c002030 con 0x7f24701457f0 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2474a9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:23.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2474a9e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.102+0000 7f2474a9e640 10 monclient: dump: 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 10 monclient: shutdown 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:23.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 1 -- 172.21.15.64:0/1111597806 >> v1:172.21.15.64:6790/0 conn(0x7f24701457f0 legacy=0x7f247013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:23.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 1 -- 172.21.15.64:0/1111597806 shutdown_connections 2024-03-29T10:52:23.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 1 -- 172.21.15.64:0/1111597806 >> 172.21.15.64:0/1111597806 conn(0x7f2470066a00 msgr2=0x7f2470066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:23.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 1 -- 172.21.15.64:0/1111597806 shutdown_connections 2024-03-29T10:52:23.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.103+0000 7f2477d04640 1 -- 172.21.15.64:0/1111597806 wait complete. 2024-03-29T10:52:23.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.104+0000 7f2477d04640 10 monclient: build_initial_monmap 2024-03-29T10:52:23.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.104+0000 7f2477d04640 10 monclient: monmap: 2024-03-29T10:52:23.093 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:23.093 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.104+0000 7f2477d04640 1 librados: starting msgr at 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.104+0000 7f2477d04640 1 librados: starting objecter 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.105+0000 7f2477d04640 1 Processor -- start 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.105+0000 7f2477d04640 1 -- start start 2024-03-29T10:52:23.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.105+0000 7f2477d04640 1 librados: setting wanted keys 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.105+0000 7f2477d04640 1 librados: calling monclient init 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.105+0000 7f2477d04640 10 monclient: init 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient(hunting): picked mon.noname-c con 0x7f24700a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient(hunting): picked mon.noname-a con 0x7f247013f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient(hunting): picked mon.noname-b con 0x7f24701457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247013aa10 con 0x7f247013f650 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247013acf0 con 0x7f24701457f0 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247013afd0 con 0x7f24700a9d70 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f2477d04640 10 monclient(hunting): authenticate will time out at 1154.405505s 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f24762a1640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f24701457f0 0x7f24701e64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:58172/0 (socket says 172.21.15.64:58172) 2024-03-29T10:52:23.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.106+0000 7f24762a1640 1 -- 172.21.15.64:0/1665260901 learned_addr learned my addr 172.21.15.64:0/1665260901 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 client.?.objecter ms_handle_connect 0x7f24701457f0 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157472920 0 0) 0x7f247013acf0 con 0x7f24701457f0 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454004eb0 con 0x7f24701457f0 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 client.?.objecter ms_handle_connect 0x7f24700a9d70 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.107+0000 7f24667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1793298134 0 0) 0x7f2454004eb0 con 0x7f24701457f0 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f247013acf0 con 0x7f24701457f0 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f24600018e0 con 0x7f24701457f0 2024-03-29T10:52:23.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): dump: 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:23.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580751517 0 0) 0x7f247013afd0 con 0x7f24700a9d70 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454004eb0 con 0x7f24700a9d70 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2060158200 0 0) 0x7f2454004eb0 con 0x7f24700a9d70 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f247013afd0 con 0x7f24700a9d70 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f246c0018d0 con 0x7f24700a9d70 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 10 monclient(hunting): dump: 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:23.098 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3309306416 0 0) 0x7f247013acf0 con 0x7f24701457f0 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 >> v1:172.21.15.64:6790/0 conn(0x7f24700a9d70 legacy=0x7f2470138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:23.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.108+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 >> v1:172.21.15.64:6789/0 conn(0x7f247013f650 legacy=0x7f2470141630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 1 monclient: found mon.b 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470139410 con 0x7f24701457f0 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 5 monclient: authenticate success, global_id 4471 2024-03-29T10:52:23.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 10 monclient: _renew_subs 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247006de90 con 0x7f24701457f0 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 10 monclient: _renew_subs 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f247013b9b0 con 0x7f24701457f0 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f2477d04640 1 librados: init done 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2460003610 con 0x7f24701457f0 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f24600039a0 con 0x7f24701457f0 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.112+0000 7f24667fc640 10 monclient: dump: 2024-03-29T10:52:23.102 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.113+0000 7f2467fff640 4 set_mon_vals no callback set 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.114+0000 7f2477d04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.114+0000 7f2477d04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.114+0000 7f2477d04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:23.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.114+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2458005080 con 0x7f24701457f0 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.114+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f2460003b00 con 0x7f24701457f0 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.115+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f2460084a90 con 0x7f24701457f0 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.115+0000 7f24667fc640 10 client.4471.objecter ms_dispatch 0x7f2470000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.115+0000 7f24667fc640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.115+0000 7f24667fc640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.115+0000 7f24667fc640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.115+0000 7f2475aa0640 10 monclient: get_auth_request con 0x7f245406dd70 auth_method 0 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.116+0000 7f24667fc640 10 client.4471.objecter ms_handle_connect 0x7f245406dd70 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.122+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2460050370 con 0x7f24701457f0 2024-03-29T10:52:23.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.122+0000 7f24667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:23.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.122+0000 7f24667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:23.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.446+0000 7f2477d04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:23.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.446+0000 7f2477d04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:23.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.446+0000 7f2477d04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:23.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.446+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2458005330 con 0x7f24701457f0 2024-03-29T10:52:23.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.448+0000 7f24667fc640 1 -- 172.21.15.64:0/1665260901 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f24600522a0 con 0x7f24701457f0 2024-03-29T10:52:23.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.448+0000 7f24667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:23.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.448+0000 7f24667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:23.430 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:23.430 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:23.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.452+0000 7f2477d04640 10 librados: watch_flush enter 2024-03-29T10:52:23.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.452+0000 7f2477d04640 10 librados: watch_flush exit 2024-03-29T10:52:23.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.452+0000 7f2477d04640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-29T10:52:23.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.452+0000 7f2477d04640 10 client.4471.objecter successfully canceled tick 2024-03-29T10:52:23.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.453+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 >> v1:172.21.15.174:6816/2473791532 conn(0x7f245406dd70 legacy=0x7f2454070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:23.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.453+0000 7f2477d04640 10 monclient: shutdown 2024-03-29T10:52:23.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.453+0000 7f2477d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:23.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.453+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 >> v1:172.21.15.174:6789/0 conn(0x7f24701457f0 legacy=0x7f24701e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:23.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.453+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 shutdown_connections 2024-03-29T10:52:23.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.453+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 >> 172.21.15.64:0/1665260901 conn(0x7f2470066a00 msgr2=0x7f24700b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:23.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.454+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 shutdown_connections 2024-03-29T10:52:23.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.454+0000 7f2477d04640 1 -- 172.21.15.64:0/1665260901 wait complete. 2024-03-29T10:52:23.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:23.454+0000 7f2477d04640 1 librados: shutdown 2024-03-29T10:52:23.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:30.261 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.435+0000 7f449fac1640 10 monclient: get_monmap_and_config 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.435+0000 7f449fac1640 10 monclient: build_initial_monmap 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.435+0000 7f449fac1640 10 monclient: monmap: 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:30.436517+0000 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:30.436517+0000 2024-03-29T10:52:30.417 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.436+0000 7f449fac1640 1 Processor -- start 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.437+0000 7f449fac1640 1 -- start start 2024-03-29T10:52:30.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.437+0000 7f449fac1640 10 monclient: init 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient(hunting): picked mon.noname-a con 0x7f44981457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient(hunting): picked mon.noname-c con 0x7f449813f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient(hunting): picked mon.noname-b con 0x7f44980a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44980aaef0 con 0x7f44981457f0 2024-03-29T10:52:30.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4498145bd0 con 0x7f449813f650 2024-03-29T10:52:30.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4498145eb0 con 0x7f44980a9d70 2024-03-29T10:52:30.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:30.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449d85d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f44981457f0 0x7f449813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57490/0 (socket says 172.21.15.64:57490) 2024-03-29T10:52:30.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449fac1640 10 monclient(hunting): authenticate will time out at 1161.737221s 2024-03-29T10:52:30.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449d85d640 1 -- 172.21.15.64:0/2294039408 learned_addr learned my addr 172.21.15.64:0/2294039408 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.438+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935812456 0 0) 0x7f4498145bd0 con 0x7f449813f650 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _negotiate 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4480003520 con 0x7f449813f650 2024-03-29T10:52:30.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357561866 0 0) 0x7f44980aaef0 con 0x7f44981457f0 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _negotiate 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4498145bd0 con 0x7f44981457f0 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143915123 0 0) 0x7f4498145eb0 con 0x7f44980a9d70 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _negotiate 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.439+0000 7f449c85b640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44980aaef0 con 0x7f44980a9d70 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288489523 0 0) 0x7f4480003520 con 0x7f449813f650 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 >> v1:172.21.15.64:6790/0 conn(0x7f44980a9d70 legacy=0x7f449814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 >> v1:172.21.15.64:6789/0 conn(0x7f44981457f0 legacy=0x7f449813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: _finish_hunting 0 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 monclient: found mon.noname-c 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4498146340 con 0x7f449813f650 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: _finish_auth 0 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: _check_auth_tickets 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449fac1640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449fac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4490001690 con 0x7f449813f650 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: dump: 2024-03-29T10:52:30.424 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449fac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44900017f0 con 0x7f449813f650 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449fac1640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449fac1640 4 set_mon_vals no callback set 2024-03-29T10:52:30.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 1 -- 172.21.15.64:0/2294039408 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4490001c00 con 0x7f449813f650 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.440+0000 7f449c85b640 10 monclient: dump: 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:30.426 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:30.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.441+0000 7f449fac1640 10 monclient: shutdown 2024-03-29T10:52:30.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.441+0000 7f449fac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:30.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.441+0000 7f449fac1640 1 -- 172.21.15.64:0/2294039408 >> v1:172.21.15.174:6789/0 conn(0x7f449813f650 legacy=0x7f44980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.442+0000 7f449fac1640 1 -- 172.21.15.64:0/2294039408 shutdown_connections 2024-03-29T10:52:30.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.442+0000 7f449fac1640 1 -- 172.21.15.64:0/2294039408 >> 172.21.15.64:0/2294039408 conn(0x7f4498066a00 msgr2=0x7f4498066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:30.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.442+0000 7f449fac1640 1 -- 172.21.15.64:0/2294039408 shutdown_connections 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.442+0000 7f449fac1640 1 -- 172.21.15.64:0/2294039408 wait complete. 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 10 monclient: build_initial_monmap 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 10 monclient: monmap: 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:30.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 1 librados: starting msgr at 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 1 librados: starting objecter 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 1 Processor -- start 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 1 -- start start 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 1 librados: setting wanted keys 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 1 librados: calling monclient init 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.443+0000 7f449fac1640 10 monclient: init 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.444+0000 7f449fac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.444+0000 7f449fac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.444+0000 7f449fac1640 10 monclient(hunting): picked mon.noname-c con 0x7f44980a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.444+0000 7f449fac1640 10 monclient(hunting): picked mon.noname-b con 0x7f449813f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.444+0000 7f449fac1640 10 monclient(hunting): picked mon.noname-a con 0x7f44981457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449fac1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44981cb940 con 0x7f44981457f0 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449fac1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44981cc7d0 con 0x7f449813f650 2024-03-29T10:52:30.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449fac1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44981ccab0 con 0x7f44980a9d70 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449fac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449fac1640 10 monclient(hunting): authenticate will time out at 1161.743919s 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449e05e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f44981457f0 0x7f44981e2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58188/0 (socket says 172.21.15.64:58188) 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f449e05e640 1 -- 172.21.15.64:0/3170354500 learned_addr learned my addr 172.21.15.64:0/3170354500 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 client.?.objecter ms_handle_connect 0x7f44980a9d70 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 client.?.objecter ms_handle_connect 0x7f44981457f0 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:30.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 client.?.objecter ms_handle_connect 0x7f449813f650 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403234003 0 0) 0x7f44981ccab0 con 0x7f44980a9d70 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4484004eb0 con 0x7f44980a9d70 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.445+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589194128 0 0) 0x7f44981cc7d0 con 0x7f449813f650 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44981ccab0 con 0x7f449813f650 2024-03-29T10:52:30.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666332896 0 0) 0x7f44981cb940 con 0x7f44981457f0 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4484005360 con 0x7f44981457f0 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934963808 0 0) 0x7f44981ccab0 con 0x7f449813f650 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44981cb940 con 0x7f449813f650 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4490001b00 con 0x7f449813f650 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): dump: 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:30.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491077286 0 0) 0x7f4484004eb0 con 0x7f44980a9d70 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.446+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44981ccab0 con 0x7f44980a9d70 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3389365102 0 0) 0x7f44981cb940 con 0x7f449813f650 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 >> v1:172.21.15.64:6790/0 conn(0x7f44980a9d70 legacy=0x7f4498138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 >> v1:172.21.15.64:6789/0 conn(0x7f44981457f0 legacy=0x7f44981e2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 1 monclient: found mon.b 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44981ccd90 con 0x7f449813f650 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 5 monclient: authenticate success, global_id 4477 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 10 monclient: _renew_subs 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44981cbdd0 con 0x7f449813f650 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 10 monclient: _renew_subs 2024-03-29T10:52:30.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44981395b0 con 0x7f449813f650 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f449fac1640 1 librados: init done 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.447+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44900038a0 con 0x7f449813f650 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.448+0000 7f44967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.448+0000 7f44977fe640 4 set_mon_vals no callback set 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.448+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4490003bb0 con 0x7f449813f650 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.448+0000 7f44967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.448+0000 7f44967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.448+0000 7f44967fc640 10 monclient: dump: 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:30.439 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.449+0000 7f449fac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.449+0000 7f449fac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.449+0000 7f449fac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.449+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4478005080 con 0x7f449813f650 2024-03-29T10:52:30.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f4490019b80 con 0x7f449813f650 2024-03-29T10:52:30.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f44900836f0 con 0x7f449813f650 2024-03-29T10:52:30.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f44967fc640 10 client.4477.objecter ms_dispatch 0x7f4498000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:30.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f44967fc640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:30.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f44967fc640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:30.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f44967fc640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T10:52:30.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.450+0000 7f449d85d640 10 monclient: get_auth_request con 0x7f448406e260 auth_method 0 2024-03-29T10:52:30.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.451+0000 7f44967fc640 10 client.4477.objecter ms_handle_connect 0x7f448406e260 2024-03-29T10:52:30.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.457+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f449004f110 con 0x7f449813f650 2024-03-29T10:52:30.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.457+0000 7f44967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:30.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.457+0000 7f44967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:30.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.890+0000 7f449fac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:30.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.890+0000 7f449fac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:30.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.890+0000 7f449fac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:30.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.890+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4478005330 con 0x7f449813f650 2024-03-29T10:52:30.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.891+0000 7f44967fc640 1 -- 172.21.15.64:0/3170354500 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f449004f110 con 0x7f449813f650 2024-03-29T10:52:30.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.891+0000 7f44967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:30.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.891+0000 7f44967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:30.873 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:30.873 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:30.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 10 librados: watch_flush enter 2024-03-29T10:52:30.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 10 librados: watch_flush exit 2024-03-29T10:52:30.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T10:52:30.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 10 client.4477.objecter successfully canceled tick 2024-03-29T10:52:30.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 >> v1:172.21.15.174:6816/2473791532 conn(0x7f448406e260 legacy=0x7f4484070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 10 monclient: shutdown 2024-03-29T10:52:30.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:30.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.895+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 >> v1:172.21.15.174:6789/0 conn(0x7f449813f650 legacy=0x7f4498137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:30.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.896+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 shutdown_connections 2024-03-29T10:52:30.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.896+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 >> 172.21.15.64:0/3170354500 conn(0x7f4498066a00 msgr2=0x7f449813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:30.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.896+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 shutdown_connections 2024-03-29T10:52:30.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.896+0000 7f449fac1640 1 -- 172.21.15.64:0/3170354500 wait complete. 2024-03-29T10:52:30.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:30.896+0000 7f449fac1640 1 librados: shutdown 2024-03-29T10:52:30.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:37.701 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.880+0000 7f764d126640 10 monclient: get_monmap_and_config 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.880+0000 7f764d126640 10 monclient: build_initial_monmap 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.880+0000 7f764d126640 10 monclient: monmap: 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:37.881228+0000 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:37.881228+0000 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:37.862 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.882+0000 7f764d126640 1 Processor -- start 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.882+0000 7f764d126640 1 -- start start 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.882+0000 7f764d126640 10 monclient: init 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient(hunting): picked mon.noname-a con 0x7f7640006e30 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient(hunting): picked mon.noname-b con 0x7f7640035e00 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient(hunting): picked mon.noname-c con 0x7f76400c9670 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7640035970 con 0x7f7640006e30 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7640036280 con 0x7f76400c9670 2024-03-29T10:52:37.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7640035640 con 0x7f7640035e00 2024-03-29T10:52:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.883+0000 7f764d126640 10 monclient(hunting): authenticate will time out at 1169.182650s 2024-03-29T10:52:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.886+0000 7f76477fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7640035e00 0x7f76400c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58812/0 (socket says 172.21.15.64:58812) 2024-03-29T10:52:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.886+0000 7f76477fe640 1 -- 172.21.15.64:0/2286628106 learned_addr learned my addr 172.21.15.64:0/2286628106 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514575980 0 0) 0x7f7640035640 con 0x7f7640035e00 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7638003520 con 0x7f7640035e00 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192173918 0 0) 0x7f7640036280 con 0x7f76400c9670 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:52:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7640035640 con 0x7f76400c9670 2024-03-29T10:52:37.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411414105 0 0) 0x7f7640035970 con 0x7f7640006e30 2024-03-29T10:52:37.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.887+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7638004a30 con 0x7f7640006e30 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705084449 0 0) 0x7f7640035640 con 0x7f76400c9670 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 >> v1:172.21.15.64:6790/0 conn(0x7f7640035e00 legacy=0x7f76400c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 >> v1:172.21.15.64:6789/0 conn(0x7f7640006e30 legacy=0x7f76400cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 monclient: found mon.noname-c 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:52:37.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7640034e20 con 0x7f76400c9670 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f764804f730 con 0x7f76400c9670 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: dump: 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.873 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f764d126640 5 monclient: authenticate success, global_id 4480 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f764d126640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7648075f10 con 0x7f76400c9670 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 1 -- 172.21.15.64:0/2286628106 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f76480758b0 con 0x7f76400c9670 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f7646ffd640 10 monclient: dump: 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:37.874 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f764d126640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.888+0000 7f764d126640 4 set_mon_vals no callback set 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 10 monclient: shutdown 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 1 -- 172.21.15.64:0/2286628106 >> v1:172.21.15.174:6789/0 conn(0x7f76400c9670 legacy=0x7f76400d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:37.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 1 -- 172.21.15.64:0/2286628106 shutdown_connections 2024-03-29T10:52:37.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 1 -- 172.21.15.64:0/2286628106 >> 172.21.15.64:0/2286628106 conn(0x7f7640005c00 msgr2=0x7f7640006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:37.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 1 -- 172.21.15.64:0/2286628106 shutdown_connections 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.890+0000 7f764d126640 1 -- 172.21.15.64:0/2286628106 wait complete. 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.891+0000 7f764d126640 10 monclient: build_initial_monmap 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.891+0000 7f764d126640 10 monclient: monmap: 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:37.877 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.891+0000 7f764d126640 1 librados: starting msgr at 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.891+0000 7f764d126640 1 librados: starting objecter 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.892+0000 7f764d126640 1 Processor -- start 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.892+0000 7f764d126640 1 -- start start 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.892+0000 7f764d126640 1 librados: setting wanted keys 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.892+0000 7f764d126640 1 librados: calling monclient init 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.892+0000 7f764d126640 10 monclient: init 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient(hunting): picked mon.noname-a con 0x7f7640006e30 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient(hunting): picked mon.noname-b con 0x7f7640035e00 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:37.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient(hunting): picked mon.noname-c con 0x7f76400c9670 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76400cd0f0 con 0x7f7640006e30 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76400ce490 con 0x7f7640035e00 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7640033060 con 0x7f76400c9670 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764d126640 10 monclient(hunting): authenticate will time out at 1169.192413s 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764c925640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f76400c9670 0x7f7640174460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38700/0 (socket says 172.21.15.64:38700) 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.893+0000 7f764c925640 1 -- 172.21.15.64:0/2568120037 learned_addr learned my addr 172.21.15.64:0/2568120037 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.898+0000 7f7644ff9640 10 client.?.objecter ms_handle_connect 0x7f76400c9670 2024-03-29T10:52:37.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.898+0000 7f7644ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:37.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 client.?.objecter ms_handle_connect 0x7f7640006e30 2024-03-29T10:52:37.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:37.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545818547 0 0) 0x7f7640033060 con 0x7f76400c9670 2024-03-29T10:52:37.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:37.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f762c004eb0 con 0x7f76400c9670 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 client.?.objecter ms_handle_connect 0x7f7640035e00 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473050562 0 0) 0x7f76400cd0f0 con 0x7f7640006e30 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7640033060 con 0x7f7640006e30 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045985964 0 0) 0x7f76400ce490 con 0x7f7640035e00 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:37.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76400cd0f0 con 0x7f7640035e00 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697734635 0 0) 0x7f76400cd0f0 con 0x7f7640035e00 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76400ce490 con 0x7f7640035e00 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7634001950 con 0x7f7640035e00 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.900+0000 7f7644ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:37.884 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569728997 0 0) 0x7f7640033060 con 0x7f7640006e30 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76400cd0f0 con 0x7f7640006e30 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f763c0018d0 con 0x7f7640006e30 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1504293965 0 0) 0x7f76400ce490 con 0x7f7640035e00 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 >> v1:172.21.15.64:6790/0 conn(0x7f76400c9670 legacy=0x7f7640174460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 >> v1:172.21.15.64:6789/0 conn(0x7f7640006e30 legacy=0x7f764016d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 monclient: found mon.b 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76400331f0 con 0x7f7640035e00 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:37.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f7644ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 10 monclient: _renew_subs 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76400339e0 con 0x7f7640035e00 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 10 monclient: _renew_subs 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76400ceb20 con 0x7f7640035e00 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.901+0000 7f764d126640 1 librados: init done 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7634001d60 con 0x7f7640035e00 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f7644ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:37.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f7634004980 con 0x7f7640035e00 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f7644ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f7644ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f7644ff9640 10 monclient: dump: 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:37.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.902+0000 7f76467fc640 4 set_mon_vals no callback set 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.903+0000 7f764d126640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.905+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f7634004ae0 con 0x7f7640035e00 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.906+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f76340849d0 con 0x7f7640035e00 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.906+0000 7f7644ff9640 10 client.4483.objecter ms_dispatch 0x7f7648000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.906+0000 7f7644ff9640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.906+0000 7f7644ff9640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.906+0000 7f7644ff9640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.907+0000 7f764d126640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.907+0000 7f764d126640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.907+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628005080 con 0x7f7640035e00 2024-03-29T10:52:37.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.908+0000 7f7647fff640 10 monclient: get_auth_request con 0x7f762c06dec0 auth_method 0 2024-03-29T10:52:37.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.908+0000 7f7644ff9640 10 client.4483.objecter ms_handle_connect 0x7f762c06dec0 2024-03-29T10:52:37.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.914+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76340502b0 con 0x7f7640035e00 2024-03-29T10:52:37.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.914+0000 7f7644ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:37.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:37.914+0000 7f7644ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:38.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.352+0000 7f764d126640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:38.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.352+0000 7f764d126640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:38.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.352+0000 7f764d126640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:52:38.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.352+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7628005330 con 0x7f7640035e00 2024-03-29T10:52:38.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.353+0000 7f7644ff9640 1 -- 172.21.15.64:0/2568120037 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f76340521e0 con 0x7f7640035e00 2024-03-29T10:52:38.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.353+0000 7f7644ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:38.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.353+0000 7f7644ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:38.335 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:38.335 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:38.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 10 librados: watch_flush enter 2024-03-29T10:52:38.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 10 librados: watch_flush exit 2024-03-29T10:52:38.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T10:52:38.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 10 client.4483.objecter successfully canceled tick 2024-03-29T10:52:38.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 >> v1:172.21.15.174:6816/2473791532 conn(0x7f762c06dec0 legacy=0x7f762c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:38.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 10 monclient: shutdown 2024-03-29T10:52:38.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:38.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 >> v1:172.21.15.174:6789/0 conn(0x7f7640035e00 legacy=0x7f764016d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:38.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 shutdown_connections 2024-03-29T10:52:38.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.358+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 >> 172.21.15.64:0/2568120037 conn(0x7f7640005c00 msgr2=0x7f76400c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:38.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.359+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 shutdown_connections 2024-03-29T10:52:38.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.359+0000 7f764d126640 1 -- 172.21.15.64:0/2568120037 wait complete. 2024-03-29T10:52:38.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:38.359+0000 7f764d126640 1 librados: shutdown 2024-03-29T10:52:38.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:45.163 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:45.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.334+0000 7f62e82d6640 10 monclient: get_monmap_and_config 2024-03-29T10:52:45.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.334+0000 7f62e82d6640 10 monclient: build_initial_monmap 2024-03-29T10:52:45.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.334+0000 7f62e82d6640 10 monclient: monmap: 2024-03-29T10:52:45.315 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:45.315 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:45.334928+0000 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:45.334928+0000 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:45.316 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.335+0000 7f62e82d6640 1 Processor -- start 2024-03-29T10:52:45.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.335+0000 7f62e82d6640 1 -- start start 2024-03-29T10:52:45.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.335+0000 7f62e82d6640 10 monclient: init 2024-03-29T10:52:45.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.336+0000 7f62e82d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.336+0000 7f62e82d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.336+0000 7f62e82d6640 10 monclient(hunting): picked mon.noname-b con 0x7f62e00b4710 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 10 monclient(hunting): picked mon.noname-c con 0x7f62e00b3970 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e6072640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f62e00b4710 0x7f62e00b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54570/0 (socket says 172.21.15.64:54570) 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e6072640 1 -- 172.21.15.64:0/514154709 learned_addr learned my addr 172.21.15.64:0/514154709 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 10 monclient(hunting): picked mon.noname-a con 0x7f62e013f390 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e00af020 con 0x7f62e013f390 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e00b0910 con 0x7f62e00b3970 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e00b0bf0 con 0x7f62e00b4710 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:45.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.337+0000 7f62e82d6640 10 monclient(hunting): authenticate will time out at 1176.636827s 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901258516 0 0) 0x7f62e00af020 con 0x7f62e013f390 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 10 monclient(hunting): _negotiate 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc003520 con 0x7f62e013f390 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540378617 0 0) 0x7f62e00b0bf0 con 0x7f62e00b4710 2024-03-29T10:52:45.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 10 monclient(hunting): _negotiate 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.338+0000 7f62e5070640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e00af020 con 0x7f62e00b4710 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3697785981 0 0) 0x7f62cc003520 con 0x7f62e013f390 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 >> v1:172.21.15.64:6790/0 conn(0x7f62e00b4710 legacy=0x7f62e00b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 >> v1:172.21.15.174:6789/0 conn(0x7f62e00b3970 legacy=0x7f62e00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 10 monclient: _finish_hunting 0 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 1 monclient: found mon.noname-a 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e00aeb00 con 0x7f62e013f390 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 10 monclient: _finish_auth 0 2024-03-29T10:52:45.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.339+0000 7f62e5070640 10 monclient: _check_auth_tickets 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e82d6640 5 monclient: authenticate success, global_id 4455 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e82d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f62d8001940 con 0x7f62e013f390 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: dump: 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:45.326 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d8001b60 con 0x7f62e013f390 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e82d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 1 -- 172.21.15.64:0/514154709 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f62d8001ef0 con 0x7f62e013f390 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e5070640 10 monclient: dump: 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.327 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e82d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.340+0000 7f62e82d6640 4 set_mon_vals no callback set 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.341+0000 7f62e82d6640 10 monclient: shutdown 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.341+0000 7f62e82d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.341+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 >> v1:172.21.15.64:6789/0 conn(0x7f62e013f390 legacy=0x7f62e013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.341+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 shutdown_connections 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.341+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 >> 172.21.15.64:0/514154709 conn(0x7f62e0066a00 msgr2=0x7f62e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:45.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.342+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 shutdown_connections 2024-03-29T10:52:45.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.342+0000 7f62e82d6640 1 -- 172.21.15.64:0/514154709 wait complete. 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.343+0000 7f62e82d6640 10 monclient: build_initial_monmap 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.343+0000 7f62e82d6640 10 monclient: monmap: 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:45.337 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.343+0000 7f62e82d6640 1 librados: starting msgr at 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.343+0000 7f62e82d6640 1 librados: starting objecter 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.344+0000 7f62e82d6640 1 Processor -- start 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.344+0000 7f62e82d6640 1 -- start start 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.344+0000 7f62e82d6640 1 librados: setting wanted keys 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.344+0000 7f62e82d6640 1 librados: calling monclient init 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.344+0000 7f62e82d6640 10 monclient: init 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e82d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e82d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e82d6640 10 monclient(hunting): picked mon.noname-c con 0x7f62e00b3970 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e6072640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f62e00b3970 0x7f62e0132bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:55120/0 (socket says 172.21.15.64:55120) 2024-03-29T10:52:45.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e6072640 1 -- 172.21.15.64:0/1710327344 learned_addr learned my addr 172.21.15.64:0/1710327344 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62d6ffd640 10 client.?.objecter ms_handle_connect 0x7f62e00b3970 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e82d6640 10 monclient(hunting): picked mon.noname-a con 0x7f62e00b4710 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e82d6640 10 monclient(hunting): picked mon.noname-b con 0x7f62e013f390 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.345+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0136060 con 0x7f62e00b4710 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 client.?.objecter ms_handle_connect 0x7f62e00b4710 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0134ad0 con 0x7f62e013f390 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 client.?.objecter ms_handle_connect 0x7f62e013f390 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0135540 con 0x7f62e00b3970 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62e82d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62e82d6640 10 monclient(hunting): authenticate will time out at 1176.645076s 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482519812 0 0) 0x7f62e0134ad0 con 0x7f62e013f390 2024-03-29T10:52:45.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c4004eb0 con 0x7f62e013f390 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322913939 0 0) 0x7f62e0136060 con 0x7f62e00b4710 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.346+0000 7f62d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e0134ad0 con 0x7f62e00b4710 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317849045 0 0) 0x7f62e0135540 con 0x7f62e00b3970 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:45.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e0136060 con 0x7f62e00b3970 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051450253 0 0) 0x7f62c4004eb0 con 0x7f62e013f390 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62e0135540 con 0x7f62e013f390 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f62d8001e70 con 0x7f62e013f390 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.341 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481130871 0 0) 0x7f62e0134ad0 con 0x7f62e00b4710 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62c4004eb0 con 0x7f62e00b4710 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f62dc001820 con 0x7f62e00b4710 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.342 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4008859325 0 0) 0x7f62e0136060 con 0x7f62e00b3970 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62e0134ad0 con 0x7f62e00b3970 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f62d0001930 con 0x7f62e00b3970 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.347+0000 7f62d6ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:45.343 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2647564925 0 0) 0x7f62c4004eb0 con 0x7f62e00b4710 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 >> v1:172.21.15.64:6790/0 conn(0x7f62e00b3970 legacy=0x7f62e0132bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 >> v1:172.21.15.174:6789/0 conn(0x7f62e013f390 legacy=0x7f62e0134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 1 monclient: found mon.a 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:45.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e01356d0 con 0x7f62e00b4710 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 10 monclient: _renew_subs 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0133300 con 0x7f62e00b4710 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 10 monclient: _renew_subs 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e0141ef0 con 0x7f62e00b4710 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.348+0000 7f62e82d6640 1 librados: init done 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62dc001cd0 con 0x7f62e00b4710 2024-03-29T10:52:45.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f62dc003790 con 0x7f62e00b4710 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62e486f640 4 set_mon_vals no callback set 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.349+0000 7f62d6ffd640 10 monclient: dump: 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:45.346 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.350+0000 7f62e82d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.350+0000 7f62e82d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.350+0000 7f62e82d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.350+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c8005080 con 0x7f62e00b4710 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.352+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f62dc019760 con 0x7f62e00b4710 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.353+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f62dc083710 con 0x7f62e00b4710 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.353+0000 7f62d6ffd640 10 client.4458.objecter ms_dispatch 0x7f62e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.353+0000 7f62d6ffd640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.353+0000 7f62d6ffd640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.353+0000 7f62d6ffd640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.353+0000 7f62e6072640 10 monclient: get_auth_request con 0x7f62c406e140 auth_method 0 2024-03-29T10:52:45.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.354+0000 7f62d6ffd640 10 client.4458.objecter ms_handle_connect 0x7f62c406e140 2024-03-29T10:52:45.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.362+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62dc04eff0 con 0x7f62e00b4710 2024-03-29T10:52:45.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.362+0000 7f62d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:45.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.362+0000 7f62d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.719+0000 7f62e82d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.719+0000 7f62e82d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.719+0000 7f62e82d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.719+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62c8005330 con 0x7f62e00b4710 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.720+0000 7f62d6ffd640 1 -- 172.21.15.64:0/1710327344 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f62dc050f20 con 0x7f62e00b4710 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.720+0000 7f62d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:45.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.720+0000 7f62d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:45.703 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:45.703 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.723+0000 7f62e82d6640 10 librados: watch_flush enter 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 10 librados: watch_flush exit 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 10 client.4458.objecter successfully canceled tick 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 >> v1:172.21.15.174:6816/2473791532 conn(0x7f62c406e140 legacy=0x7f62c4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 10 monclient: shutdown 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 >> v1:172.21.15.64:6789/0 conn(0x7f62e00b4710 legacy=0x7f62e0133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 shutdown_connections 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 >> 172.21.15.64:0/1710327344 conn(0x7f62e0066a00 msgr2=0x7f62e00b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:45.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 shutdown_connections 2024-03-29T10:52:45.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.724+0000 7f62e82d6640 1 -- 172.21.15.64:0/1710327344 wait complete. 2024-03-29T10:52:45.707 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:45.725+0000 7f62e82d6640 1 librados: shutdown 2024-03-29T10:52:45.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:52.530 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:52.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.684+0000 7f46ac6a1640 10 monclient: get_monmap_and_config 2024-03-29T10:52:52.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.684+0000 7f46ac6a1640 10 monclient: build_initial_monmap 2024-03-29T10:52:52.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.684+0000 7f46ac6a1640 10 monclient: monmap: 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:52.684998+0000 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:52.684998+0000 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:52.666 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:52.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.684+0000 7f46ac6a1640 1 Processor -- start 2024-03-29T10:52:52.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.684+0000 7f46ac6a1640 1 -- start start 2024-03-29T10:52:52.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.685+0000 7f46ac6a1640 10 monclient: init 2024-03-29T10:52:52.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.685+0000 7f46ac6a1640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:52.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.685+0000 7f46ac6a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 10 monclient(hunting): picked mon.noname-a con 0x7f46a41457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 10 monclient(hunting): picked mon.noname-c con 0x7f46a413f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 10 monclient(hunting): picked mon.noname-b con 0x7f46a40a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40aaef0 con 0x7f46a41457f0 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a4145bd0 con 0x7f46a413f650 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a4145eb0 con 0x7f46a40a9d70 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46ac6a1640 10 monclient(hunting): authenticate will time out at 1183.984935s 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46aac3e640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f46a40a9d70 0x7f46a414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52892/0 (socket says 172.21.15.64:52892) 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46aac3e640 1 -- 172.21.15.64:0/3466067512 learned_addr learned my addr 172.21.15.64:0/3466067512 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917568791 0 0) 0x7f46a4145eb0 con 0x7f46a40a9d70 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 10 monclient(hunting): _negotiate 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:52.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:52.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:52:52.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690003520 con 0x7f46a40a9d70 2024-03-29T10:52:52.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000689243 0 0) 0x7f4690003520 con 0x7f46a40a9d70 2024-03-29T10:52:52.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 >> v1:172.21.15.174:6789/0 conn(0x7f46a413f650 legacy=0x7f46a40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:52.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.686+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 >> v1:172.21.15.64:6789/0 conn(0x7f46a41457f0 legacy=0x7f46a413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: _finish_hunting 0 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 1 monclient: found mon.noname-b 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a4146340 con 0x7f46a40a9d70 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: _finish_auth 0 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: _check_auth_tickets 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f469c001a10 con 0x7f46a40a9d70 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:52.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 10 monclient: dump: 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:52.673 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46ac6a1640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46ac6a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.689+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f469c001be0 con 0x7f46a40a9d70 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46a943b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46a943b640 1 -- 172.21.15.64:0/3466067512 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f469c002030 con 0x7f46a40a9d70 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46ac6a1640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:52.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46ac6a1640 4 set_mon_vals no callback set 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46a943b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46a943b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46a943b640 10 monclient: dump: 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:52.675 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46ac6a1640 10 monclient: shutdown 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46ac6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:52.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.690+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3466067512 >> v1:172.21.15.64:6790/0 conn(0x7f46a40a9d70 legacy=0x7f46a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:52.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.691+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3466067512 shutdown_connections 2024-03-29T10:52:52.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.691+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3466067512 >> 172.21.15.64:0/3466067512 conn(0x7f46a4066a00 msgr2=0x7f46a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:52.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.692+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3466067512 shutdown_connections 2024-03-29T10:52:52.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.692+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3466067512 wait complete. 2024-03-29T10:52:52.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.693+0000 7f46ac6a1640 10 monclient: build_initial_monmap 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.693+0000 7f46ac6a1640 10 monclient: monmap: 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.693+0000 7f46ac6a1640 1 librados: starting msgr at 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.694+0000 7f46ac6a1640 1 librados: starting objecter 2024-03-29T10:52:52.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.694+0000 7f46ac6a1640 1 Processor -- start 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.694+0000 7f46ac6a1640 1 -- start start 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.694+0000 7f46ac6a1640 1 librados: setting wanted keys 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.694+0000 7f46ac6a1640 1 librados: calling monclient init 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.694+0000 7f46ac6a1640 10 monclient: init 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient(hunting): picked mon.noname-c con 0x7f46a40a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46aa43d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f46a40a9d70 0x7f46a41cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53314/0 (socket says 172.21.15.64:53314) 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient(hunting): picked mon.noname-b con 0x7f46a413f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient(hunting): picked mon.noname-a con 0x7f46a41457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41cb490 con 0x7f46a41457f0 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46aa43d640 1 -- 172.21.15.64:0/3909316522 learned_addr learned my addr 172.21.15.64:0/3909316522 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:52.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41387a0 con 0x7f46a413f650 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f469affd640 10 client.?.objecter ms_handle_connect 0x7f46a40a9d70 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f469affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41cc9b0 con 0x7f46a40a9d70 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f46ac6a1640 10 monclient(hunting): authenticate will time out at 1183.994660s 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887904779 0 0) 0x7f46a41cc9b0 con 0x7f46a40a9d70 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f469affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.695+0000 7f469affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4688004eb0 con 0x7f46a40a9d70 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733131726 0 0) 0x7f4688004eb0 con 0x7f46a40a9d70 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4688005300 con 0x7f46a40a9d70 2024-03-29T10:52:52.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f46a0001790 con 0x7f46a40a9d70 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 10 monclient(hunting): dump: 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:52.688 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 965676644 0 0) 0x7f4688005300 con 0x7f46a40a9d70 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 >> v1:172.21.15.174:6789/0 conn(0x7f46a413f650 legacy=0x7f46a4137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.696+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 >> v1:172.21.15.64:6789/0 conn(0x7f46a41457f0 legacy=0x7f46a4138090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 1 monclient: found mon.c 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a41ccc90 con 0x7f46a40a9d70 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 10 monclient: _finish_auth 0 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f469affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f46ac6a1640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f46ac6a1640 10 monclient: _renew_subs 2024-03-29T10:52:52.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f46ac6a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.697+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a41ccf70 con 0x7f46a40a9d70 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a0001d60 con 0x7f46a40a9d70 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f469affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f46a8c3a640 4 set_mon_vals no callback set 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f46a0001790 con 0x7f46a40a9d70 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f469affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f469affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.698+0000 7f469affd640 10 monclient: dump: 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:52.690 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.699+0000 7f46ac6a1640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.699+0000 7f46ac6a1640 10 monclient: _renew_subs 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.699+0000 7f46ac6a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.699+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a41cbb00 con 0x7f46a40a9d70 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.699+0000 7f46ac6a1640 1 librados: init done 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.700+0000 7f46ac6a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.700+0000 7f46ac6a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.700+0000 7f46ac6a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.700+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a41cb240 con 0x7f46a40a9d70 2024-03-29T10:52:52.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f46a001a750 con 0x7f46a40a9d70 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f46a00845a0 con 0x7f46a40a9d70 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f469affd640 10 client.4502.objecter ms_dispatch 0x7f46a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f469affd640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f469affd640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f469affd640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.701+0000 7f46a9c3c640 10 monclient: get_auth_request con 0x7f468806e220 auth_method 0 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.702+0000 7f469affd640 10 client.4502.objecter ms_handle_connect 0x7f468806e220 2024-03-29T10:52:52.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.707+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46a004fe80 con 0x7f46a40a9d70 2024-03-29T10:52:52.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.707+0000 7f469affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:52.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:52.707+0000 7f469affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:52.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.014+0000 7f46ac6a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:52.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.015+0000 7f46ac6a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:52.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.015+0000 7f46ac6a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:52:52.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.015+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46a4075f10 con 0x7f46a40a9d70 2024-03-29T10:52:52.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.016+0000 7f469affd640 1 -- 172.21.15.64:0/3909316522 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f46a00015f0 con 0x7f46a40a9d70 2024-03-29T10:52:52.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.016+0000 7f469affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:52.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.016+0000 7f469affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:52.997 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:52:52.997 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 10 librados: watch_flush enter 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 10 librados: watch_flush exit 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 10 client.4502.objecter successfully canceled tick 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 >> v1:172.21.15.174:6816/2473791532 conn(0x7f468806e220 legacy=0x7f46880706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 10 monclient: shutdown 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:53.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.019+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 >> v1:172.21.15.64:6790/0 conn(0x7f46a40a9d70 legacy=0x7f46a41cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.020+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 shutdown_connections 2024-03-29T10:52:53.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.020+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 >> 172.21.15.64:0/3909316522 conn(0x7f46a4066a00 msgr2=0x7f46a40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:53.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.020+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 shutdown_connections 2024-03-29T10:52:53.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.020+0000 7f46ac6a1640 1 -- 172.21.15.64:0/3909316522 wait complete. 2024-03-29T10:52:53.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:53.020+0000 7f46ac6a1640 1 librados: shutdown 2024-03-29T10:52:53.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:59.823 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:59.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.982+0000 7fd5a2520640 10 monclient: get_monmap_and_config 2024-03-29T10:52:59.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.982+0000 7fd5a2520640 10 monclient: build_initial_monmap 2024-03-29T10:52:59.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.982+0000 7fd5a2520640 10 monclient: monmap: 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:52:59.984247+0000 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:52:59.984247+0000 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:52:59.965 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:59.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.984+0000 7fd5a2520640 1 Processor -- start 2024-03-29T10:52:59.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.985+0000 7fd5a2520640 1 -- start start 2024-03-29T10:52:59.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.985+0000 7fd5a2520640 10 monclient: init 2024-03-29T10:52:59.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:59.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:59.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient(hunting): picked mon.noname-b con 0x7fd594006e30 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:59.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient(hunting): picked mon.noname-c con 0x7fd594035e00 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:59.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient(hunting): picked mon.noname-a con 0x7fd5940c9670 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:59.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd594035970 con 0x7fd5940c9670 2024-03-29T10:52:59.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd594036280 con 0x7fd594035e00 2024-03-29T10:52:59.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd594035640 con 0x7fd594006e30 2024-03-29T10:52:59.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:59.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.986+0000 7fd5a2520640 10 monclient(hunting): authenticate will time out at 1191.286235s 2024-03-29T10:52:59.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.989+0000 7fd5a0d1d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fd594035e00 0x7fd5940c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:34758/0 (socket says 172.21.15.64:34758) 2024-03-29T10:52:59.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.989+0000 7fd5a0d1d640 1 -- 172.21.15.64:0/1739813101 learned_addr learned my addr 172.21.15.64:0/1739813101 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:59.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.989+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818683046 0 0) 0x7fd594036280 con 0x7fd594035e00 2024-03-29T10:52:59.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.989+0000 7fd593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:59.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.989+0000 7fd593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:59.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.990+0000 7fd593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:59.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.990+0000 7fd593fff640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:52:59.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.990+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd588003520 con 0x7fd594035e00 2024-03-29T10:52:59.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.990+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433759760 0 0) 0x7fd594035640 con 0x7fd594006e30 2024-03-29T10:52:59.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.990+0000 7fd593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.990+0000 7fd593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd588003840 con 0x7fd594006e30 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931477740 0 0) 0x7fd594035970 con 0x7fd5940c9670 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:59.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd594035640 con 0x7fd5940c9670 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552855266 0 0) 0x7fd588003520 con 0x7fd594035e00 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 >> v1:172.21.15.64:6790/0 conn(0x7fd594006e30 legacy=0x7fd5940cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 >> v1:172.21.15.64:6789/0 conn(0x7fd5940c9670 legacy=0x7fd5940d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 monclient: found mon.noname-c 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594034e20 con 0x7fd594035e00 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:59.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd5980015f0 con 0x7fd594035e00 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: dump: 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:59.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd5a2520640 5 monclient: authenticate success, global_id 4492 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd5a2520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd598001750 con 0x7fd594035e00 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 1 -- 172.21.15.64:0/1739813101 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd598001ba0 con 0x7fd594035e00 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd593fff640 10 monclient: dump: 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:59.979 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd5a2520640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.991+0000 7fd5a2520640 4 set_mon_vals no callback set 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.992+0000 7fd5a2520640 10 monclient: shutdown 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.992+0000 7fd5a2520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.992+0000 7fd5a2520640 1 -- 172.21.15.64:0/1739813101 >> v1:172.21.15.174:6789/0 conn(0x7fd594035e00 legacy=0x7fd5940c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.993+0000 7fd5a2520640 1 -- 172.21.15.64:0/1739813101 shutdown_connections 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.993+0000 7fd5a2520640 1 -- 172.21.15.64:0/1739813101 >> 172.21.15.64:0/1739813101 conn(0x7fd594005c00 msgr2=0x7fd594006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.993+0000 7fd5a2520640 1 -- 172.21.15.64:0/1739813101 shutdown_connections 2024-03-29T10:52:59.980 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.993+0000 7fd5a2520640 1 -- 172.21.15.64:0/1739813101 wait complete. 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.994+0000 7fd5a2520640 10 monclient: build_initial_monmap 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.994+0000 7fd5a2520640 10 monclient: monmap: 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:52:59.994 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 1 librados: starting msgr at 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 1 librados: starting objecter 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 1 Processor -- start 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 1 -- start start 2024-03-29T10:52:59.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 1 librados: setting wanted keys 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 1 librados: calling monclient init 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.995+0000 7fd5a2520640 10 monclient: init 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.996+0000 7fd5a2520640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.996+0000 7fd5a2520640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 10 monclient(hunting): picked mon.noname-a con 0x7fd594006e30 addr v1:172.21.15.64:6789/0 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 10 monclient(hunting): picked mon.noname-c con 0x7fd594035e00 addr v1:172.21.15.64:6790/0 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 10 monclient(hunting): picked mon.noname-b con 0x7fd5940c9670 addr v1:172.21.15.174:6789/0 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59402f150 con 0x7fd594006e30 2024-03-29T10:52:59.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59402f430 con 0x7fd5940c9670 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59402e230 con 0x7fd594035e00 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.997+0000 7fd5a2520640 10 monclient(hunting): authenticate will time out at 1191.297621s 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd5a151e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd594006e30 0x7fd59402fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51023/0 (socket says 172.21.15.64:51023) 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd5a151e640 1 -- 172.21.15.64:0/872232823 learned_addr learned my addr 172.21.15.64:0/872232823 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd591ffb640 10 client.?.objecter ms_handle_connect 0x7fd594006e30 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd591ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991506348 0 0) 0x7fd59402f150 con 0x7fd594006e30 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd591ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd591ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:59.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:52:59.999+0000 7fd591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.000+0000 7fd591ffb640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.000+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580004eb0 con 0x7fd594006e30 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.000+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1141923243 0 0) 0x7fd580004eb0 con 0x7fd594006e30 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd59402f150 con 0x7fd594006e30 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd58c001a30 con 0x7fd594006e30 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:59.998 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 client.?.objecter ms_handle_connect 0x7fd594035e00 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284376959 0 0) 0x7fd59402e230 con 0x7fd594035e00 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.001+0000 7fd591ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.002+0000 7fd591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.002+0000 7fd591ffb640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.002+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580004eb0 con 0x7fd594035e00 2024-03-29T10:52:59.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.002+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1707602079 0 0) 0x7fd59402f150 con 0x7fd594006e30 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 >> v1:172.21.15.64:6790/0 conn(0x7fd594035e00 legacy=0x7fd594164930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 >> v1:172.21.15.174:6789/0 conn(0x7fd5940c9670 legacy=0x7fd59416ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 1 monclient: found mon.a 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.003+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59402e3c0 con 0x7fd594006e30 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 client.?.objecter ms_handle_connect 0x7fd5940c9670 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd58c001d30 con 0x7fd594006e30 2024-03-29T10:53:00.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fd58c003a60 con 0x7fd594006e30 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd592ffd640 4 set_mon_vals no callback set 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd591ffb640 10 monclient: dump: 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:00.001 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.004+0000 7fd5a2520640 5 monclient: authenticate success, global_id 4464 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 10 monclient: _renew_subs 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd594165040 con 0x7fd594006e30 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 10 monclient: _renew_subs 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59416d1b0 con 0x7fd594006e30 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.005+0000 7fd5a2520640 1 librados: init done 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.007+0000 7fd5a2520640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.007+0000 7fd5a2520640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:00.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.007+0000 7fd5a2520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.007+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd594007b90 con 0x7fd594006e30 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fd58c003bc0 con 0x7fd594006e30 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fd58c084bd0 con 0x7fd594006e30 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd591ffb640 10 client.4464.objecter ms_dispatch 0x7fd59c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd591ffb640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd591ffb640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd591ffb640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.008+0000 7fd5a0d1d640 10 monclient: get_auth_request con 0x7fd58006dd40 auth_method 0 2024-03-29T10:53:00.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.010+0000 7fd591ffb640 10 client.4464.objecter ms_handle_connect 0x7fd58006dd40 2024-03-29T10:53:00.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.018+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd58c0502e0 con 0x7fd594006e30 2024-03-29T10:53:00.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.018+0000 7fd591ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:00.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.018+0000 7fd591ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:00.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd5a2520640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:00.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd5a2520640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd5a2520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd594033ee0 con 0x7fd594006e30 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd591ffb640 1 -- 172.21.15.64:0/872232823 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fd58c0523e0 con 0x7fd594006e30 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd591ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.361+0000 7fd591ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:00.344 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.364+0000 7fd5a2520640 10 librados: watch_flush enter 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.364+0000 7fd5a2520640 10 librados: watch_flush exit 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.364+0000 7fd5a2520640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.364+0000 7fd5a2520640 10 client.4464.objecter successfully canceled tick 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.365+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 >> v1:172.21.15.174:6816/2473791532 conn(0x7fd58006dd40 legacy=0x7fd580070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.365+0000 7fd5a2520640 10 monclient: shutdown 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.365+0000 7fd5a2520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:00.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.365+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 >> v1:172.21.15.64:6789/0 conn(0x7fd594006e30 legacy=0x7fd59402fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:00.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.365+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 shutdown_connections 2024-03-29T10:53:00.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.365+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 >> 172.21.15.64:0/872232823 conn(0x7fd594005c00 msgr2=0x7fd594005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:00.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.367+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 shutdown_connections 2024-03-29T10:53:00.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.367+0000 7fd5a2520640 1 -- 172.21.15.64:0/872232823 wait complete. 2024-03-29T10:53:00.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:00.368+0000 7fd5a2520640 1 librados: shutdown 2024-03-29T10:53:00.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:07.175 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.338+0000 7f46e4863640 10 monclient: get_monmap_and_config 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.338+0000 7f46e4863640 10 monclient: build_initial_monmap 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.338+0000 7f46e4863640 10 monclient: monmap: 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:07.340062+0000 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:07.340062+0000 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:07.322 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.342+0000 7f46e4863640 1 Processor -- start 2024-03-29T10:53:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.342+0000 7f46e4863640 1 -- start start 2024-03-29T10:53:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.342+0000 7f46e4863640 10 monclient: init 2024-03-29T10:53:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient(hunting): picked mon.noname-c con 0x7f46e00a9750 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient(hunting): picked mon.noname-b con 0x7f46e00adca0 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient(hunting): picked mon.noname-a con 0x7f46e0146a80 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00abdc0 con 0x7f46e0146a80 2024-03-29T10:53:07.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00b0070 con 0x7f46e00a9750 2024-03-29T10:53:07.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00af880 con 0x7f46e00adca0 2024-03-29T10:53:07.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:07.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.343+0000 7f46e4863640 10 monclient(hunting): authenticate will time out at 1198.643241s 2024-03-29T10:53:07.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.346+0000 7f46ded9d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f46e00a9750 0x7f46e0143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:39014/0 (socket says 172.21.15.64:39014) 2024-03-29T10:53:07.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.346+0000 7f46ded9d640 1 -- 172.21.15.64:0/813299890 learned_addr learned my addr 172.21.15.64:0/813299890 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:07.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798752223 0 0) 0x7f46e00abdc0 con 0x7f46e0146a80 2024-03-29T10:53:07.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:53:07.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:07.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:07.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:53:07.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c8003520 con 0x7f46e0146a80 2024-03-29T10:53:07.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 349671648 0 0) 0x7f46c8003520 con 0x7f46e0146a80 2024-03-29T10:53:07.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 >> v1:172.21.15.64:6790/0 conn(0x7f46e00adca0 legacy=0x7f46e0146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:07.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 >> v1:172.21.15.174:6789/0 conn(0x7f46e00a9750 legacy=0x7f46e0143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 monclient: found mon.noname-a 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e00ac030 con 0x7f46e0146a80 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: _finish_auth 0 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:07.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46e4863640 5 monclient: authenticate success, global_id 4467 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46e4863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f46d0001670 con 0x7f46e0146a80 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46ddd9b640 10 monclient: dump: 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:07.332 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:07.333 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:07.333 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:07.333 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:07.333 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:07.334 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:07.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.347+0000 7f46e4863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.348+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d00018d0 con 0x7f46e0146a80 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.348+0000 7f46ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.348+0000 7f46e4863640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.348+0000 7f46e4863640 4 set_mon_vals no callback set 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.349+0000 7f46e4863640 10 monclient: shutdown 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.349+0000 7f46e4863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.349+0000 7f46e4863640 1 -- 172.21.15.64:0/813299890 >> v1:172.21.15.64:6789/0 conn(0x7f46e0146a80 legacy=0x7f46e0151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.349+0000 7f46ddd9b640 1 -- 172.21.15.64:0/813299890 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f46d0001670 con 0x7f46e0146a80 2024-03-29T10:53:07.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.349+0000 7f46ddd9b640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.351+0000 7f46e4863640 1 -- 172.21.15.64:0/813299890 shutdown_connections 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.351+0000 7f46e4863640 1 -- 172.21.15.64:0/813299890 >> 172.21.15.64:0/813299890 conn(0x7f46e00673e0 msgr2=0x7f46e00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.351+0000 7f46e4863640 1 -- 172.21.15.64:0/813299890 shutdown_connections 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.352+0000 7f46e4863640 1 -- 172.21.15.64:0/813299890 wait complete. 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.353+0000 7f46e4863640 10 monclient: build_initial_monmap 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.353+0000 7f46e4863640 10 monclient: monmap: 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:07.341 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.353+0000 7f46e4863640 1 librados: starting msgr at 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.353+0000 7f46e4863640 1 librados: starting objecter 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 Processor -- start 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 -- start start 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 librados: setting wanted keys 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 librados: calling monclient init 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient: init 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient(hunting): picked mon.noname-c con 0x7f46e00a9750 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient(hunting): picked mon.noname-a con 0x7f46e00adca0 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient(hunting): picked mon.noname-b con 0x7f46e0146a80 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:07.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00b1880 con 0x7f46e00adca0 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e01e4af0 con 0x7f46e0146a80 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00b2f50 con 0x7f46e00a9750 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.354+0000 7f46e4863640 10 monclient(hunting): authenticate will time out at 1198.654812s 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.356+0000 7f46df59e640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f46e0146a80 0x7f46e01e43e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:41174/0 (socket says 172.21.15.64:41174) 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.356+0000 7f46df59e640 1 -- 172.21.15.64:0/3964599651 learned_addr learned my addr 172.21.15.64:0/3964599651 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 10 client.?.objecter ms_handle_connect 0x7f46e0146a80 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:07.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369783692 0 0) 0x7f46e01e4af0 con 0x7f46e0146a80 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.357+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46cc004eb0 con 0x7f46e0146a80 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 10 client.?.objecter ms_handle_connect 0x7f46e00adca0 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715743434 0 0) 0x7f46e00b1880 con 0x7f46e00adca0 2024-03-29T10:53:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:07.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:07.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:07.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:53:07.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e01e4af0 con 0x7f46e00adca0 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2501461194 0 0) 0x7f46e01e4af0 con 0x7f46e00adca0 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.358+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e00b1880 con 0x7f46e00adca0 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f46d4001830 con 0x7f46e00adca0 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient(hunting): dump: 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:07.351 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 client.?.objecter ms_handle_connect 0x7f46e00a9750 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997974904 0 0) 0x7f46e00b1880 con 0x7f46e00adca0 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 >> v1:172.21.15.64:6790/0 conn(0x7f46e00a9750 legacy=0x7f46e0144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 >> v1:172.21.15.174:6789/0 conn(0x7f46e0146a80 legacy=0x7f46e01e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 1 monclient: found mon.a 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:07.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e00b3230 con 0x7f46e00adca0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 5 monclient: authenticate success, global_id 4470 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 10 monclient: _renew_subs 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e006d960 con 0x7f46e00adca0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 10 monclient: _renew_subs 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e0144880 con 0x7f46e00adca0 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.359+0000 7f46e4863640 1 librados: init done 2024-03-29T10:53:07.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.361+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d4001d40 con 0x7f46e00adca0 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.361+0000 7f46c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.361+0000 7f46dcd99640 4 set_mon_vals no callback set 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.362+0000 7f46e4863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.362+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f46d4001830 con 0x7f46e00adca0 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.362+0000 7f46c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.362+0000 7f46c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.362+0000 7f46c77fe640 10 monclient: dump: 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:07.354 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:07.355 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:07.355 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:07.355 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.363+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f46d4019560 con 0x7f46e00adca0 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.363+0000 7f46e4863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.363+0000 7f46e4863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.363+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c0005080 con 0x7f46e00adca0 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.364+0000 7f46ded9d640 10 monclient: get_auth_request con 0x7f46cc06def0 auth_method 0 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.364+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f46d4053350 con 0x7f46e00adca0 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.364+0000 7f46c77fe640 10 client.4470.objecter ms_dispatch 0x7f46e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.364+0000 7f46c77fe640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.364+0000 7f46c77fe640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:07.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.364+0000 7f46c77fe640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T10:53:07.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.365+0000 7f46c77fe640 10 client.4470.objecter ms_handle_connect 0x7f46cc06def0 2024-03-29T10:53:07.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.376+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46d404f5f0 con 0x7f46e00adca0 2024-03-29T10:53:07.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.376+0000 7f46c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:07.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.376+0000 7f46c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:07.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.706+0000 7f46e4863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:07.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.706+0000 7f46e4863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:07.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.706+0000 7f46e4863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:07.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.706+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46c0005330 con 0x7f46e00adca0 2024-03-29T10:53:07.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.709+0000 7f46c77fe640 1 -- 172.21.15.64:0/3964599651 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f46d404f5f0 con 0x7f46e00adca0 2024-03-29T10:53:07.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.709+0000 7f46c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:07.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.709+0000 7f46c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:07.692 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:07.692 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:07.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 10 librados: watch_flush enter 2024-03-29T10:53:07.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 10 librados: watch_flush exit 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 10 client.4470.objecter successfully canceled tick 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 >> v1:172.21.15.174:6816/2473791532 conn(0x7f46cc06def0 legacy=0x7f46cc0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 10 monclient: shutdown 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 >> v1:172.21.15.64:6789/0 conn(0x7f46e00adca0 legacy=0x7f46e01e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 shutdown_connections 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.713+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 >> 172.21.15.64:0/3964599651 conn(0x7f46e00673e0 msgr2=0x7f46e00ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.714+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 shutdown_connections 2024-03-29T10:53:07.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.714+0000 7f46e4863640 1 -- 172.21.15.64:0/3964599651 wait complete. 2024-03-29T10:53:07.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:07.714+0000 7f46e4863640 1 librados: shutdown 2024-03-29T10:53:07.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:14.520 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:14.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.690+0000 7f4928a80640 10 monclient: get_monmap_and_config 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.690+0000 7f4928a80640 10 monclient: build_initial_monmap 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.690+0000 7f4928a80640 10 monclient: monmap: 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:14.691466+0000 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:14.691466+0000 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:14.672 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:14.673 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:14.673 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:14.673 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:14.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.695+0000 7f4928a80640 1 Processor -- start 2024-03-29T10:53:14.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.695+0000 7f4928a80640 1 -- start start 2024-03-29T10:53:14.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.695+0000 7f4928a80640 10 monclient: init 2024-03-29T10:53:14.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient(hunting): picked mon.noname-c con 0x7f491c006b00 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient(hunting): picked mon.noname-b con 0x7f491c033000 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient(hunting): picked mon.noname-a con 0x7f491c02c230 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c02b7e0 con 0x7f491c02c230 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c02d210 con 0x7f491c006b00 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c02d4f0 con 0x7f491c033000 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:14.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.696+0000 7f4928a80640 10 monclient(hunting): authenticate will time out at 1205.995169s 2024-03-29T10:53:14.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.700+0000 7f49237fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f491c006b00 0x7f491c0c15d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:40241/0 (socket says 172.21.15.64:40241) 2024-03-29T10:53:14.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.700+0000 7f49237fe640 1 -- 172.21.15.64:0/212691296 learned_addr learned my addr 172.21.15.64:0/212691296 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260766832 0 0) 0x7f491c02d210 con 0x7f491c006b00 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914005730 con 0x7f491c006b00 2024-03-29T10:53:14.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711907749 0 0) 0x7f4914005730 con 0x7f491c006b00 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 >> v1:172.21.15.64:6790/0 conn(0x7f491c033000 legacy=0x7f491c0c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 >> v1:172.21.15.64:6789/0 conn(0x7f491c02c230 legacy=0x7f491c0cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 monclient: found mon.noname-c 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.701+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c02b2a0 con 0x7f491c006b00 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f4928a80640 5 monclient: authenticate success, global_id 4501 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f4928a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4910001690 con 0x7f491c006b00 2024-03-29T10:53:14.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 10 monclient: dump: 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:14.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f4928a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 1 -- 172.21.15.64:0/212691296 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49100017f0 con 0x7f491c006b00 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f49227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f4928a80640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.702+0000 7f4928a80640 4 set_mon_vals no callback set 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.703+0000 7f4928a80640 10 monclient: shutdown 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.703+0000 7f4928a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.703+0000 7f4928a80640 1 -- 172.21.15.64:0/212691296 >> v1:172.21.15.174:6789/0 conn(0x7f491c006b00 legacy=0x7f491c0c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.703+0000 7f4928a80640 1 -- 172.21.15.64:0/212691296 shutdown_connections 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.703+0000 7f4928a80640 1 -- 172.21.15.64:0/212691296 >> 172.21.15.64:0/212691296 conn(0x7f491c0058f0 msgr2=0x7f491c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.704+0000 7f4928a80640 1 -- 172.21.15.64:0/212691296 shutdown_connections 2024-03-29T10:53:14.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.706+0000 7f4928a80640 1 -- 172.21.15.64:0/212691296 wait complete. 2024-03-29T10:53:14.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 10 monclient: build_initial_monmap 2024-03-29T10:53:14.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 10 monclient: monmap: 2024-03-29T10:53:14.688 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:14.688 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:14.688 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:14.688 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 1 librados: starting msgr at 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 1 librados: starting objecter 2024-03-29T10:53:14.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 1 Processor -- start 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 1 -- start start 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 1 librados: setting wanted keys 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 1 librados: calling monclient init 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.707+0000 7f4928a80640 10 monclient: init 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient(hunting): picked mon.noname-c con 0x7f491c006b00 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient(hunting): picked mon.noname-b con 0x7f491c02c230 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient(hunting): picked mon.noname-a con 0x7f491c0c7800 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:14.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c0c51d0 con 0x7f491c0c7800 2024-03-29T10:53:14.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c163d60 con 0x7f491c02c230 2024-03-29T10:53:14.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c163ef0 con 0x7f491c006b00 2024-03-29T10:53:14.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:14.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.708+0000 7f4928a80640 10 monclient(hunting): authenticate will time out at 1206.007736s 2024-03-29T10:53:14.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.711+0000 7f49237fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f491c006b00 0x7f491c0c4970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56629/0 (socket says 172.21.15.64:56629) 2024-03-29T10:53:14.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.711+0000 7f49237fe640 1 -- 172.21.15.64:0/270581835 learned_addr learned my addr 172.21.15.64:0/270581835 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:14.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 10 client.?.objecter ms_handle_connect 0x7f491c006b00 2024-03-29T10:53:14.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:14.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614043375 0 0) 0x7f491c163ef0 con 0x7f491c006b00 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4908004eb0 con 0x7f491c006b00 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.712+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026594347 0 0) 0x7f4908004eb0 con 0x7f491c006b00 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f491c163ef0 con 0x7f491c006b00 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4910004ae0 con 0x7f491c006b00 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 10 monclient(hunting): dump: 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:14.695 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1408809547 0 0) 0x7f491c163ef0 con 0x7f491c006b00 2024-03-29T10:53:14.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.713+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 >> v1:172.21.15.174:6789/0 conn(0x7f491c02c230 legacy=0x7f491c038320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 >> v1:172.21.15.64:6789/0 conn(0x7f491c0c7800 legacy=0x7f491c0bb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 1 monclient: found mon.c 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c0c2470 con 0x7f491c006b00 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:14.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.715+0000 7f490ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 10 monclient: _renew_subs 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491c0c2720 con 0x7f491c006b00 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 10 monclient: _renew_subs 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c0c2c50 con 0x7f491c006b00 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4928a80640 1 librados: init done 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4910005060 con 0x7f491c006b00 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f490ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:14.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f4921ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:14.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.716+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4910005470 con 0x7f491c006b00 2024-03-29T10:53:14.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.717+0000 7f490ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:14.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.717+0000 7f490ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:14.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.717+0000 7f490ffff640 10 monclient: dump: 2024-03-29T10:53:14.702 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:14.702 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:14.702 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:14.702 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:14.703 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:14.703 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:14.703 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:14.703 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:14.703 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:14.703 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:14.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.720+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f4910001aa0 con 0x7f491c006b00 2024-03-29T10:53:14.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f4922ffd640 10 monclient: get_auth_request con 0x7f490806dcd0 auth_method 0 2024-03-29T10:53:14.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f4910084fc0 con 0x7f491c006b00 2024-03-29T10:53:14.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f490ffff640 10 client.4514.objecter ms_dispatch 0x7f4924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:14.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f490ffff640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:14.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f490ffff640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:14.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f490ffff640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T10:53:14.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.721+0000 7f490ffff640 10 client.4514.objecter ms_handle_connect 0x7f490806dcd0 2024-03-29T10:53:14.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.724+0000 7f4928a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:14.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.724+0000 7f4928a80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:14.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.724+0000 7f4928a80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:14.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.724+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904005080 con 0x7f491c006b00 2024-03-29T10:53:14.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.734+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49100508a0 con 0x7f491c006b00 2024-03-29T10:53:14.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.734+0000 7f490ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:14.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:14.734+0000 7f490ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:15.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.053+0000 7f4928a80640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:15.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.053+0000 7f4928a80640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:15.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.053+0000 7f4928a80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:15.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.053+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4904005330 con 0x7f491c006b00 2024-03-29T10:53:15.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.057+0000 7f490ffff640 1 -- 172.21.15.64:0/270581835 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f49100527d0 con 0x7f491c006b00 2024-03-29T10:53:15.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.057+0000 7f490ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:15.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.057+0000 7f490ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:15.040 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:15.041 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:15.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 10 librados: watch_flush enter 2024-03-29T10:53:15.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 10 librados: watch_flush exit 2024-03-29T10:53:15.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T10:53:15.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 10 client.4514.objecter successfully canceled tick 2024-03-29T10:53:15.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 >> v1:172.21.15.174:6816/2473791532 conn(0x7f490806dcd0 legacy=0x7f4908070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:15.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 10 monclient: shutdown 2024-03-29T10:53:15.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:15.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.063+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 >> v1:172.21.15.64:6790/0 conn(0x7f491c006b00 legacy=0x7f491c0c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:15.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.064+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 shutdown_connections 2024-03-29T10:53:15.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.064+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 >> 172.21.15.64:0/270581835 conn(0x7f491c0058f0 msgr2=0x7f491c14ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:15.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.067+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 shutdown_connections 2024-03-29T10:53:15.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.067+0000 7f4928a80640 1 -- 172.21.15.64:0/270581835 wait complete. 2024-03-29T10:53:15.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:15.067+0000 7f4928a80640 1 librados: shutdown 2024-03-29T10:53:15.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:21.877 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:22.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.043+0000 7f16b2c55640 10 monclient: get_monmap_and_config 2024-03-29T10:53:22.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.043+0000 7f16b2c55640 10 monclient: build_initial_monmap 2024-03-29T10:53:22.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.043+0000 7f16b2c55640 10 monclient: monmap: 2024-03-29T10:53:22.024 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:22.024 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:22.024 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:22.043936+0000 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:22.043936+0000 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:22.025 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:22.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.046+0000 7f16b2c55640 1 Processor -- start 2024-03-29T10:53:22.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.046+0000 7f16b2c55640 1 -- start start 2024-03-29T10:53:22.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.046+0000 7f16b2c55640 10 monclient: init 2024-03-29T10:53:22.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:22.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:22.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient(hunting): picked mon.noname-b con 0x7f16ac0b1f60 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:22.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient(hunting): picked mon.noname-a con 0x7f16ac071eb0 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:22.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient(hunting): picked mon.noname-c con 0x7f16ac143d10 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:22.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac06ef80 con 0x7f16ac071eb0 2024-03-29T10:53:22.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac06f260 con 0x7f16ac143d10 2024-03-29T10:53:22.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0b40c0 con 0x7f16ac0b1f60 2024-03-29T10:53:22.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:22.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.047+0000 7f16b2c55640 10 monclient(hunting): authenticate will time out at 1213.345987s 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16b09f1640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f16ac0b1f60 0x7f16ac143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53113/0 (socket says 172.21.15.64:53113) 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16b09f1640 1 -- 172.21.15.64:0/2973912389 learned_addr learned my addr 172.21.15.64:0/2973912389 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226889840 0 0) 0x7f16ac0b40c0 con 0x7f16ac0b1f60 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:22.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1694003520 con 0x7f16ac0b1f60 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 596045678 0 0) 0x7f1694003520 con 0x7f16ac0b1f60 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 >> v1:172.21.15.174:6789/0 conn(0x7f16ac143d10 legacy=0x7f16ac146b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 >> v1:172.21.15.64:6789/0 conn(0x7f16ac071eb0 legacy=0x7f16ac146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 monclient: found mon.noname-b 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac155d70 con 0x7f16ac0b1f60 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:22.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f16a0001690 con 0x7f16ac0b1f60 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: dump: 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:22.042 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16a0001860 con 0x7f16ac0b1f60 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 1 -- 172.21.15.64:0/2973912389 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f16a0001cb0 con 0x7f16ac0b1f60 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.052+0000 7f16ab7fe640 10 monclient: dump: 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:22.043 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:22.044 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.044 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.044 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:22.044 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:22.044 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:22.044 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.053+0000 7f16b2c55640 5 monclient: authenticate success, global_id 4517 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.053+0000 7f16b2c55640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.053+0000 7f16b2c55640 4 set_mon_vals no callback set 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.054+0000 7f16b2c55640 10 monclient: shutdown 2024-03-29T10:53:22.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.054+0000 7f16b2c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.054+0000 7f16b2c55640 1 -- 172.21.15.64:0/2973912389 >> v1:172.21.15.64:6790/0 conn(0x7f16ac0b1f60 legacy=0x7f16ac143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.054+0000 7f16b2c55640 1 -- 172.21.15.64:0/2973912389 shutdown_connections 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.055+0000 7f16b2c55640 1 -- 172.21.15.64:0/2973912389 >> 172.21.15.64:0/2973912389 conn(0x7f16ac066a00 msgr2=0x7f16ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.055+0000 7f16b2c55640 1 -- 172.21.15.64:0/2973912389 shutdown_connections 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.055+0000 7f16b2c55640 1 -- 172.21.15.64:0/2973912389 wait complete. 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.056+0000 7f16b2c55640 10 monclient: build_initial_monmap 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.056+0000 7f16b2c55640 10 monclient: monmap: 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:22.046 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.056+0000 7f16b2c55640 1 librados: starting msgr at 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.056+0000 7f16b2c55640 1 librados: starting objecter 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.057+0000 7f16b2c55640 1 Processor -- start 2024-03-29T10:53:22.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.057+0000 7f16b2c55640 1 -- start start 2024-03-29T10:53:22.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.057+0000 7f16b2c55640 1 librados: setting wanted keys 2024-03-29T10:53:22.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.057+0000 7f16b2c55640 1 librados: calling monclient init 2024-03-29T10:53:22.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.057+0000 7f16b2c55640 10 monclient: init 2024-03-29T10:53:22.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.060+0000 7f16b2c55640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:22.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.060+0000 7f16b2c55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.060+0000 7f16b2c55640 10 monclient(hunting): picked mon.noname-b con 0x7f16ac0b1f60 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.062+0000 7f16b2c55640 10 monclient(hunting): picked mon.noname-c con 0x7f16ac143d10 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.063+0000 7f16b2c55640 10 monclient(hunting): picked mon.noname-a con 0x7f16ac06f620 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.063+0000 7f16b2c55640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac1329c0 con 0x7f16ac06f620 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.063+0000 7f16b2c55640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac141f80 con 0x7f16ac0b1f60 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.063+0000 7f16b2c55640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac142bb0 con 0x7f16ac143d10 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.063+0000 7f16b2c55640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.063+0000 7f16b2c55640 10 monclient(hunting): authenticate will time out at 1213.362813s 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.065+0000 7f16abfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f16ac143d10 0x7f16ac137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53199/0 (socket says 172.21.15.64:53199) 2024-03-29T10:53:22.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.065+0000 7f16abfff640 1 -- 172.21.15.64:0/1151193313 learned_addr learned my addr 172.21.15.64:0/1151193313 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 10 client.?.objecter ms_handle_connect 0x7f16ac143d10 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373020522 0 0) 0x7f16ac142bb0 con 0x7f16ac143d10 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.066+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1698004eb0 con 0x7f16ac143d10 2024-03-29T10:53:22.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.067+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1352029645 0 0) 0x7f1698004eb0 con 0x7f16ac143d10 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.067+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16ac142bb0 con 0x7f16ac143d10 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.067+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f169c0018b0 con 0x7f16ac143d10 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.067+0000 7f16a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient(hunting): dump: 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:22.053 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 542055503 0 0) 0x7f16ac142bb0 con 0x7f16ac143d10 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 >> v1:172.21.15.174:6789/0 conn(0x7f16ac0b1f60 legacy=0x7f16ac1424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 >> v1:172.21.15.64:6789/0 conn(0x7f16ac06f620 legacy=0x7f16ac137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:53:22.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 1 monclient: found mon.c 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac142d40 con 0x7f16ac143d10 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient: _finish_auth 0 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.068+0000 7f16a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 5 monclient: authenticate success, global_id 4520 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 10 monclient: _renew_subs 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac141b50 con 0x7f16ac143d10 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:22.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 10 monclient: _renew_subs 2024-03-29T10:53:22.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:22.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac132e40 con 0x7f16ac143d10 2024-03-29T10:53:22.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.069+0000 7f16b2c55640 1 librados: init done 2024-03-29T10:53:22.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.070+0000 7f16b2c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16b2c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16b2c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac142190 con 0x7f16ac143d10 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f169c001d00 con 0x7f16ac143d10 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f169c001b10 con 0x7f16ac143d10 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 10 monclient: dump: 2024-03-29T10:53:22.059 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.071+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f169c003960 con 0x7f16ac143d10 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f169c0849a0 con 0x7f16ac143d10 2024-03-29T10:53:22.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16a97fa640 10 client.4520.objecter ms_dispatch 0x7f16ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:22.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16a97fa640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:22.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16a97fa640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:22.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16a97fa640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T10:53:22.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16aaffd640 4 set_mon_vals no callback set 2024-03-29T10:53:22.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.072+0000 7f16b09f1640 10 monclient: get_auth_request con 0x7f169806dc80 auth_method 0 2024-03-29T10:53:22.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.076+0000 7f16a97fa640 10 client.4520.objecter ms_handle_connect 0x7f169806dc80 2024-03-29T10:53:22.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.081+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f169c050150 con 0x7f16ac143d10 2024-03-29T10:53:22.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.081+0000 7f16a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:22.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.081+0000 7f16a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:22.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.407+0000 7f16b2c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:22.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.407+0000 7f16b2c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:22.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.407+0000 7f16b2c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:22.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.407+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ac141d90 con 0x7f16ac143d10 2024-03-29T10:53:22.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.408+0000 7f16a97fa640 1 -- 172.21.15.64:0/1151193313 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f169c052250 con 0x7f16ac143d10 2024-03-29T10:53:22.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.408+0000 7f16a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:22.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.408+0000 7f16a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:22.391 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:22.391 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:22.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.411+0000 7f16b2c55640 10 librados: watch_flush enter 2024-03-29T10:53:22.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 10 librados: watch_flush exit 2024-03-29T10:53:22.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T10:53:22.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 10 client.4520.objecter successfully canceled tick 2024-03-29T10:53:22.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 >> v1:172.21.15.174:6816/2473791532 conn(0x7f169806dc80 legacy=0x7f1698070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:22.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 10 monclient: shutdown 2024-03-29T10:53:22.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:22.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.412+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 >> v1:172.21.15.64:6790/0 conn(0x7f16ac143d10 legacy=0x7f16ac137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:22.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.416+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 shutdown_connections 2024-03-29T10:53:22.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.416+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 >> 172.21.15.64:0/1151193313 conn(0x7f16ac066a00 msgr2=0x7f16ac146af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:22.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.417+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 shutdown_connections 2024-03-29T10:53:22.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.417+0000 7f16b2c55640 1 -- 172.21.15.64:0/1151193313 wait complete. 2024-03-29T10:53:22.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:22.418+0000 7f16b2c55640 1 librados: shutdown 2024-03-29T10:53:22.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:29.224 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.391+0000 7fe1e340f640 10 monclient: get_monmap_and_config 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.391+0000 7fe1e340f640 10 monclient: build_initial_monmap 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.391+0000 7fe1e340f640 10 monclient: monmap: 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:29.392476+0000 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:29.392476+0000 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:29.373 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:29.374 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.392+0000 7fe1e340f640 1 Processor -- start 2024-03-29T10:53:29.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.393+0000 7fe1e340f640 1 -- start start 2024-03-29T10:53:29.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.393+0000 7fe1e340f640 10 monclient: init 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient(hunting): picked mon.noname-b con 0x7fe1dc1457f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient(hunting): picked mon.noname-a con 0x7fe1dc13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient(hunting): picked mon.noname-c con 0x7fe1dc0a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc0aaef0 con 0x7fe1dc13f650 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc145bd0 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc145eb0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:29.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.395+0000 7fe1e340f640 10 monclient(hunting): authenticate will time out at 1220.694530s 2024-03-29T10:53:29.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1e09aa640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe1dc13f650 0x7fe1dc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57400/0 (socket says 172.21.15.64:57400) 2024-03-29T10:53:29.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1e09aa640 1 -- 172.21.15.64:0/223383438 learned_addr learned my addr 172.21.15.64:0/223383438 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:29.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896163251 0 0) 0x7fe1dc0aaef0 con 0x7fe1dc13f650 2024-03-29T10:53:29.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:29.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:29.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:29.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1d3fff640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:53:29.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.397+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c8003520 con 0x7fe1dc13f650 2024-03-29T10:53:29.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.398+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549062039 0 0) 0x7fe1c8003520 con 0x7fe1dc13f650 2024-03-29T10:53:29.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.398+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 >> v1:172.21.15.64:6790/0 conn(0x7fe1dc1457f0 legacy=0x7fe1dc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:29.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.398+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 >> v1:172.21.15.174:6789/0 conn(0x7fe1dc0a9d70 legacy=0x7fe1dc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:29.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:29.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 1 monclient: found mon.noname-a 2024-03-29T10:53:29.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc146340 con 0x7fe1dc13f650 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1d8001690 con 0x7fe1dc13f650 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1d3fff640 10 monclient: dump: 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.384 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1e340f640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.400+0000 7fe1e340f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d8001860 con 0x7fe1dc13f650 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1d3fff640 1 -- 172.21.15.64:0/223383438 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1d8001cb0 con 0x7fe1dc13f650 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1d3fff640 10 monclient: dump: 2024-03-29T10:53:29.385 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:29.386 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1e340f640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:29.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.402+0000 7fe1e340f640 4 set_mon_vals no callback set 2024-03-29T10:53:29.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.404+0000 7fe1e340f640 10 monclient: shutdown 2024-03-29T10:53:29.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.404+0000 7fe1e340f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:29.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.404+0000 7fe1e340f640 1 -- 172.21.15.64:0/223383438 >> v1:172.21.15.64:6789/0 conn(0x7fe1dc13f650 legacy=0x7fe1dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:29.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.405+0000 7fe1e340f640 1 -- 172.21.15.64:0/223383438 shutdown_connections 2024-03-29T10:53:29.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.405+0000 7fe1e340f640 1 -- 172.21.15.64:0/223383438 >> 172.21.15.64:0/223383438 conn(0x7fe1dc066a00 msgr2=0x7fe1dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:29.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.407+0000 7fe1e340f640 1 -- 172.21.15.64:0/223383438 shutdown_connections 2024-03-29T10:53:29.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.408+0000 7fe1e340f640 1 -- 172.21.15.64:0/223383438 wait complete. 2024-03-29T10:53:29.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.408+0000 7fe1e340f640 10 monclient: build_initial_monmap 2024-03-29T10:53:29.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.409+0000 7fe1e340f640 10 monclient: monmap: 2024-03-29T10:53:29.390 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.409+0000 7fe1e340f640 1 librados: starting msgr at 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.409+0000 7fe1e340f640 1 librados: starting objecter 2024-03-29T10:53:29.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.409+0000 7fe1e340f640 1 Processor -- start 2024-03-29T10:53:29.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.410+0000 7fe1e340f640 1 -- start start 2024-03-29T10:53:29.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.410+0000 7fe1e340f640 1 librados: setting wanted keys 2024-03-29T10:53:29.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.410+0000 7fe1e340f640 1 librados: calling monclient init 2024-03-29T10:53:29.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.410+0000 7fe1e340f640 10 monclient: init 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.411+0000 7fe1e340f640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.411+0000 7fe1e340f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.411+0000 7fe1e340f640 10 monclient(hunting): picked mon.noname-b con 0x7fe1dc0a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.411+0000 7fe1e340f640 10 monclient(hunting): picked mon.noname-c con 0x7fe1dc13f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.414+0000 7fe1e09aa640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe1dc13f650 0x7fe1dc135500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39047/0 (socket says 172.21.15.64:39047) 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.414+0000 7fe1e19ac640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe1dc1457f0 0x7fe1dc135c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:32912/0 (socket says 172.21.15.64:32912) 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e11ab640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe1dc0a9d70 0x7fe1dc1413c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:40439/0 (socket says 172.21.15.64:40439) 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e340f640 10 monclient(hunting): picked mon.noname-a con 0x7fe1dc1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e340f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc133150 con 0x7fe1dc1457f0 2024-03-29T10:53:29.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e09aa640 1 -- 172.21.15.64:0/474418874 learned_addr learned my addr 172.21.15.64:0/474418874 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe1dc1457f0 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe1dc0a9d70 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe1dc13f650 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40592613 0 0) 0x7fe1dc133150 con 0x7fe1dc1457f0 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc134b20 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc134e00 con 0x7fe1dc13f650 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e340f640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.415+0000 7fe1e340f640 10 monclient(hunting): authenticate will time out at 1220.714812s 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:29.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c0004eb0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951876607 0 0) 0x7fe1dc134b20 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc133150 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141332272 0 0) 0x7fe1dc134e00 con 0x7fe1dc13f650 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:53:29.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc134b20 con 0x7fe1dc13f650 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464938760 0 0) 0x7fe1c0004eb0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.416+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc134e00 con 0x7fe1dc1457f0 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1cc0017f0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.402 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2271935257 0 0) 0x7fe1dc133150 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1c0004eb0 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993217515 0 0) 0x7fe1dc134b20 con 0x7fe1dc13f650 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc133150 con 0x7fe1dc13f650 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1d4001a50 con 0x7fe1dc0a9d70 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1d8001b70 con 0x7fe1dc13f650 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2069966088 0 0) 0x7fe1dc134e00 con 0x7fe1dc1457f0 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 >> v1:172.21.15.64:6790/0 conn(0x7fe1dc13f650 legacy=0x7fe1dc135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 >> v1:172.21.15.174:6789/0 conn(0x7fe1dc0a9d70 legacy=0x7fe1dc1413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 monclient: found mon.a 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc1350e0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.417+0000 7fe1d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.418+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1cc001950 con 0x7fe1dc1457f0 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.418+0000 7fe1d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.418+0000 7fe1d37fe640 4 set_mon_vals no callback set 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe1cc0049e0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1d1ffb640 10 monclient: dump: 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1e340f640 5 monclient: authenticate success, global_id 4476 2024-03-29T10:53:29.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1e340f640 10 monclient: _renew_subs 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1e340f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.419+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc133dd0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.420+0000 7fe1e340f640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.420+0000 7fe1e340f640 10 monclient: _renew_subs 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.420+0000 7fe1e340f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.420+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc141cb0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.420+0000 7fe1e340f640 1 librados: init done 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.422+0000 7fe1e340f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.422+0000 7fe1e340f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.422+0000 7fe1e340f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.422+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc075f10 con 0x7fe1dc1457f0 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.422+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe1cc01a9b0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.423+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe1cc0848f0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.423+0000 7fe1d1ffb640 10 client.4476.objecter ms_dispatch 0x7fe1dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.423+0000 7fe1d1ffb640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.423+0000 7fe1d1ffb640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.423+0000 7fe1d1ffb640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.424+0000 7fe1e11ab640 10 monclient: get_auth_request con 0x7fe1c006e280 auth_method 0 2024-03-29T10:53:29.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.425+0000 7fe1d1ffb640 10 client.4476.objecter ms_handle_connect 0x7fe1c006e280 2024-03-29T10:53:29.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.431+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1cc0501d0 con 0x7fe1dc1457f0 2024-03-29T10:53:29.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.431+0000 7fe1d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:29.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.431+0000 7fe1d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:29.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.767+0000 7fe1e340f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:29.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.767+0000 7fe1e340f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:29.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.767+0000 7fe1e340f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:53:29.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.767+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1dc0b3680 con 0x7fe1dc1457f0 2024-03-29T10:53:29.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.768+0000 7fe1d1ffb640 1 -- 172.21.15.64:0/474418874 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe1cc052100 con 0x7fe1dc1457f0 2024-03-29T10:53:29.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.768+0000 7fe1d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:29.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.768+0000 7fe1d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:29.749 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:29.750 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:29.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.771+0000 7fe1e340f640 10 librados: watch_flush enter 2024-03-29T10:53:29.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.771+0000 7fe1e340f640 10 librados: watch_flush exit 2024-03-29T10:53:29.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.771+0000 7fe1e340f640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T10:53:29.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.771+0000 7fe1e340f640 10 client.4476.objecter successfully canceled tick 2024-03-29T10:53:29.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.772+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe1c006e280 legacy=0x7fe1c0070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:29.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.772+0000 7fe1e340f640 10 monclient: shutdown 2024-03-29T10:53:29.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.772+0000 7fe1e340f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:29.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.772+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 >> v1:172.21.15.64:6789/0 conn(0x7fe1dc1457f0 legacy=0x7fe1dc135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:29.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.772+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 shutdown_connections 2024-03-29T10:53:29.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.772+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 >> 172.21.15.64:0/474418874 conn(0x7fe1dc066a00 msgr2=0x7fe1dc130a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:29.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.774+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 shutdown_connections 2024-03-29T10:53:29.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.774+0000 7fe1e340f640 1 -- 172.21.15.64:0/474418874 wait complete. 2024-03-29T10:53:29.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:29.774+0000 7fe1e340f640 1 librados: shutdown 2024-03-29T10:53:29.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:36.581 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.755+0000 7f5610fab640 10 monclient: get_monmap_and_config 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.755+0000 7f5610fab640 10 monclient: build_initial_monmap 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.756+0000 7f5610fab640 10 monclient: monmap: 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:36.757180+0000 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:36.757180+0000 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:36.737 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:36.738 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:36.738 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:36.738 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:36.738 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:36.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.757+0000 7f5610fab640 1 Processor -- start 2024-03-29T10:53:36.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.757+0000 7f5610fab640 1 -- start start 2024-03-29T10:53:36.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.757+0000 7f5610fab640 10 monclient: init 2024-03-29T10:53:36.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.758+0000 7f5610fab640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 10 monclient(hunting): picked mon.noname-b con 0x7f560c1418f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 10 monclient(hunting): picked mon.noname-a con 0x7f560c06f5f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 10 monclient(hunting): picked mon.noname-c con 0x7f560c13ba50 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f560a59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f560c1418f0 0x7f560c141f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51722/0 (socket says 172.21.15.64:51722) 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f560a59c640 1 -- 172.21.15.64:0/2263018629 learned_addr learned my addr 172.21.15.64:0/2263018629 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c142640 con 0x7f560c06f5f0 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c06e930 con 0x7f560c13ba50 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.759+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0707f0 con 0x7f560c1418f0 2024-03-29T10:53:36.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.760+0000 7f5610fab640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:36.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.760+0000 7f5610fab640 10 monclient(hunting): authenticate will time out at 1228.059027s 2024-03-29T10:53:36.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.760+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438333293 0 0) 0x7f560c06e930 con 0x7f560c13ba50 2024-03-29T10:53:36.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:36.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:36.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:36.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f8003440 con 0x7f560c13ba50 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136607817 0 0) 0x7f560c142640 con 0x7f560c06f5f0 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c06e930 con 0x7f560c06f5f0 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255431982 0 0) 0x7f560c0707f0 con 0x7f560c1418f0 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:53:36.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.761+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c142640 con 0x7f560c1418f0 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212246024 0 0) 0x7f55f8003440 con 0x7f560c13ba50 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 >> v1:172.21.15.64:6790/0 conn(0x7f560c1418f0 legacy=0x7f560c141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 >> v1:172.21.15.64:6789/0 conn(0x7f560c06f5f0 legacy=0x7f560c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: _finish_hunting 0 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 monclient: found mon.noname-c 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c06eb80 con 0x7f560c13ba50 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: _finish_auth 0 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: _check_auth_tickets 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f55fc001610 con 0x7f560c13ba50 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:36.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: dump: 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f5610fab640 5 monclient: authenticate success, global_id 4507 2024-03-29T10:53:36.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f5610fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55fc001770 con 0x7f560c13ba50 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 1 -- 172.21.15.64:0/2263018629 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f55fc001bc0 con 0x7f560c13ba50 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f560959a640 10 monclient: dump: 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:36.751 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.762+0000 7f5610fab640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.763+0000 7f5610fab640 4 set_mon_vals no callback set 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 10 monclient: shutdown 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 >> v1:172.21.15.174:6789/0 conn(0x7f560c13ba50 legacy=0x7f560c149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 shutdown_connections 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 >> 172.21.15.64:0/2263018629 conn(0x7f560c0b3e60 msgr2=0x7f560c0b4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 shutdown_connections 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.764+0000 7f5610fab640 1 -- 172.21.15.64:0/2263018629 wait complete. 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.765+0000 7f5610fab640 10 monclient: build_initial_monmap 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.765+0000 7f5610fab640 10 monclient: monmap: 2024-03-29T10:53:36.752 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 1 librados: starting msgr at 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 1 librados: starting objecter 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 1 Processor -- start 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 1 -- start start 2024-03-29T10:53:36.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 1 librados: setting wanted keys 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 1 librados: calling monclient init 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.766+0000 7f5610fab640 10 monclient: init 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.767+0000 7f5610fab640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.767+0000 7f5610fab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.767+0000 7f5610fab640 10 monclient(hunting): picked mon.noname-c con 0x7f560c06f5f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.767+0000 7f5610fab640 10 monclient(hunting): picked mon.noname-a con 0x7f560c13ba50 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.767+0000 7f5610fab640 10 monclient(hunting): picked mon.noname-b con 0x7f560c1418f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:36.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5610fab640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0a9880 con 0x7f560c13ba50 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5610fab640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0aa720 con 0x7f560c1418f0 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5610fab640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0aaa00 con 0x7f560c06f5f0 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5610fab640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5610fab640 10 monclient(hunting): authenticate will time out at 1228.066959s 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f560a59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f560c06f5f0 0x7f560c13bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52332/0 (socket says 172.21.15.64:52332) 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f560a59c640 1 -- 172.21.15.64:0/2216399414 learned_addr learned my addr 172.21.15.64:0/2216399414 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 client.?.objecter ms_handle_connect 0x7f560c06f5f0 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292776508 0 0) 0x7f560c0aaa00 con 0x7f560c06f5f0 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:36.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ec004eb0 con 0x7f560c06f5f0 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 client.?.objecter ms_handle_connect 0x7f560c1418f0 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.768+0000 7f5602ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 10 client.?.objecter ms_handle_connect 0x7f560c13ba50 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327050677 0 0) 0x7f55ec004eb0 con 0x7f560c06f5f0 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f560c0aaa00 con 0x7f560c06f5f0 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f56040013a0 con 0x7f560c06f5f0 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.761 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.769+0000 7f5602ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648608316 0 0) 0x7f560c0aa720 con 0x7f560c1418f0 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ec004eb0 con 0x7f560c1418f0 2024-03-29T10:53:36.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989558722 0 0) 0x7f560c0a9880 con 0x7f560c13ba50 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c0aa720 con 0x7f560c13ba50 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3640886790 0 0) 0x7f560c0aaa00 con 0x7f560c06f5f0 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 >> v1:172.21.15.174:6789/0 conn(0x7f560c1418f0 legacy=0x7f560c1e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 >> v1:172.21.15.64:6789/0 conn(0x7f560c13ba50 legacy=0x7f560c1dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 monclient: found mon.c 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c135920 con 0x7f560c06f5f0 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:36.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5602ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 10 monclient: _renew_subs 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c13eb90 con 0x7f560c06f5f0 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 10 monclient: _renew_subs 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c1411e0 con 0x7f560c06f5f0 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.770+0000 7f5610fab640 1 librados: init done 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56040018d0 con 0x7f560c06f5f0 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5602ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5603fff640 4 set_mon_vals no callback set 2024-03-29T10:53:36.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5604001ce0 con 0x7f560c06f5f0 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5602ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5602ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.771+0000 7f5602ffd640 10 monclient: dump: 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.772+0000 7f5600ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:36.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.772+0000 7f5600ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.772+0000 7f5600ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.772+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e4005080 con 0x7f560c06f5f0 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.774+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f5604001e40 con 0x7f560c06f5f0 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f5604084d80 con 0x7f560c06f5f0 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5609d9b640 10 monclient: get_auth_request con 0x7f55ec06df20 auth_method 0 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5602ffd640 10 client.4529.objecter ms_dispatch 0x7f560c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5602ffd640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5602ffd640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5602ffd640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T10:53:36.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.775+0000 7f5602ffd640 10 client.4529.objecter ms_handle_connect 0x7f55ec06df20 2024-03-29T10:53:36.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.782+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5604050660 con 0x7f560c06f5f0 2024-03-29T10:53:36.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.782+0000 7f5602ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:36.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:36.782+0000 7f5602ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:37.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.131+0000 7f5600ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:37.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.132+0000 7f5600ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:37.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.132+0000 7f5600ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:37.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.132+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55e4005330 con 0x7f560c06f5f0 2024-03-29T10:53:37.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.132+0000 7f5602ffd640 1 -- 172.21.15.64:0/2216399414 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f5604052590 con 0x7f560c06f5f0 2024-03-29T10:53:37.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.132+0000 7f5602ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:37.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.132+0000 7f5602ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:37.114 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:37.114 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.135+0000 7f5600ff9640 10 librados: watch_flush enter 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.135+0000 7f5600ff9640 10 librados: watch_flush exit 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.135+0000 7f5600ff9640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.135+0000 7f5600ff9640 10 client.4529.objecter successfully canceled tick 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.136+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 >> v1:172.21.15.174:6816/2473791532 conn(0x7f55ec06df20 legacy=0x7f55ec0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.136+0000 7f5600ff9640 10 monclient: shutdown 2024-03-29T10:53:37.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.136+0000 7f5600ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:37.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.136+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 >> v1:172.21.15.64:6790/0 conn(0x7f560c06f5f0 legacy=0x7f560c13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:37.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.136+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 shutdown_connections 2024-03-29T10:53:37.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.136+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 >> 172.21.15.64:0/2216399414 conn(0x7f560c0b3e60 msgr2=0x7f560c0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:37.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.138+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 shutdown_connections 2024-03-29T10:53:37.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.138+0000 7f5600ff9640 1 -- 172.21.15.64:0/2216399414 wait complete. 2024-03-29T10:53:37.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:37.138+0000 7f5600ff9640 1 librados: shutdown 2024-03-29T10:53:37.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:43.941 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.125+0000 7fe881c4e640 10 monclient: get_monmap_and_config 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.125+0000 7fe881c4e640 10 monclient: build_initial_monmap 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.125+0000 7fe881c4e640 10 monclient: monmap: 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:44.126501+0000 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:44.126501+0000 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.107 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:44.108 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:44.108 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:44.108 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.126+0000 7fe881c4e640 1 Processor -- start 2024-03-29T10:53:44.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.127+0000 7fe881c4e640 1 -- start start 2024-03-29T10:53:44.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.127+0000 7fe881c4e640 10 monclient: init 2024-03-29T10:53:44.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.127+0000 7fe881c4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:44.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.127+0000 7fe881c4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:44.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fe87c0b1f60 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:44.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fe87c071eb0 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fe87c143d10 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c06ef80 con 0x7fe87c143d10 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c06f260 con 0x7fe87c0b1f60 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c0b40c0 con 0x7fe87c071eb0 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe881c4e640 10 monclient(hunting): authenticate will time out at 1235.426949s 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe87bfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe87c143d10 0x7fe87c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42706/0 (socket says 172.21.15.64:42706) 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe87bfff640 1 -- 172.21.15.64:0/762391165 learned_addr learned my addr 172.21.15.64:0/762391165 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:44.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.128+0000 7fe87affd640 1 --1- 172.21.15.64:0/762391165 >> v1:172.21.15.64:6790/0 conn(0x7fe87c071eb0 0x7fe87c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39120/0 (socket says 172.21.15.64:39120) 2024-03-29T10:53:44.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806219345 0 0) 0x7fe87c0b40c0 con 0x7fe87c071eb0 2024-03-29T10:53:44.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe864003520 con 0x7fe87c071eb0 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512258516 0 0) 0x7fe87c06f260 con 0x7fe87c0b1f60 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c0b40c0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069478254 0 0) 0x7fe87c06ef80 con 0x7fe87c143d10 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c06f260 con 0x7fe87c143d10 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3759702753 0 0) 0x7fe87c0b40c0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 >> v1:172.21.15.64:6790/0 conn(0x7fe87c071eb0 legacy=0x7fe87c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 >> v1:172.21.15.64:6789/0 conn(0x7fe87c143d10 legacy=0x7fe87c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:53:44.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c155d70 con 0x7fe87c0b1f60 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe881c4e640 5 monclient: authenticate success, global_id 4513 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe881c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe86c0015f0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.129+0000 7fe87a7fc640 10 monclient: dump: 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.115 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe881c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe86c001750 con 0x7fe87c0b1f60 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe87a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe881c4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe881c4e640 4 set_mon_vals no callback set 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe87a7fc640 1 -- 172.21.15.64:0/762391165 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe86c001b60 con 0x7fe87c0b1f60 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe87a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:44.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe87a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.130+0000 7fe87a7fc640 10 monclient: dump: 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 10 monclient: shutdown 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:44.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 1 -- 172.21.15.64:0/762391165 >> v1:172.21.15.174:6789/0 conn(0x7fe87c0b1f60 legacy=0x7fe87c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 1 -- 172.21.15.64:0/762391165 shutdown_connections 2024-03-29T10:53:44.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 1 -- 172.21.15.64:0/762391165 >> 172.21.15.64:0/762391165 conn(0x7fe87c066a00 msgr2=0x7fe87c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:44.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 1 -- 172.21.15.64:0/762391165 shutdown_connections 2024-03-29T10:53:44.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.131+0000 7fe881c4e640 1 -- 172.21.15.64:0/762391165 wait complete. 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.132+0000 7fe881c4e640 10 monclient: build_initial_monmap 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.132+0000 7fe881c4e640 10 monclient: monmap: 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:44.121 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 1 librados: starting msgr at 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 1 librados: starting objecter 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 1 Processor -- start 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 1 -- start start 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 1 librados: setting wanted keys 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 1 librados: calling monclient init 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.133+0000 7fe881c4e640 10 monclient: init 2024-03-29T10:53:44.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fe87c0b1f60 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fe87c143d10 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fe87c149340 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c149800 con 0x7fe87c143d10 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c14b1e0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c14b4c0 con 0x7fe87c149340 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe881c4e640 10 monclient(hunting): authenticate will time out at 1235.433278s 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe87b7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe87c0b1f60 0x7fe87c070690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:46436/0 (socket says 172.21.15.64:46436) 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe87b7fe640 1 -- 172.21.15.64:0/97930741 learned_addr learned my addr 172.21.15.64:0/97930741 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe863fff640 10 client.?.objecter ms_handle_connect 0x7fe87c0b1f60 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.134+0000 7fe863fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:44.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 client.?.objecter ms_handle_connect 0x7fe87c149340 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 client.?.objecter ms_handle_connect 0x7fe87c143d10 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980774265 0 0) 0x7fe87c14b1e0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe868004eb0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603571256 0 0) 0x7fe87c14b4c0 con 0x7fe87c149340 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:44.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.135+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c14b1e0 con 0x7fe87c149340 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1859539294 0 0) 0x7fe87c14b1e0 con 0x7fe87c149340 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe87c14b4c0 con 0x7fe87c149340 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe874001a70 con 0x7fe87c149340 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): dump: 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.125 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736152380 0 0) 0x7fe868004eb0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe87c14b1e0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302784876 0 0) 0x7fe87c149800 con 0x7fe87c143d10 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe868004eb0 con 0x7fe87c143d10 2024-03-29T10:53:44.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe86c001ae0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.136+0000 7fe863fff640 10 monclient(hunting): dump: 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:44.127 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1689011044 0 0) 0x7fe87c14b1e0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 >> v1:172.21.15.64:6790/0 conn(0x7fe87c149340 legacy=0x7fe87c1e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 >> v1:172.21.15.64:6789/0 conn(0x7fe87c143d10 legacy=0x7fe87c147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 1 monclient: found mon.b 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c149a50 con 0x7fe87c0b1f60 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe863fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:44.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 10 monclient: _renew_subs 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c147ea0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 10 monclient: _renew_subs 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c14a150 con 0x7fe87c0b1f60 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.137+0000 7fe881c4e640 1 librados: init done 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.139+0000 7fe881c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.139+0000 7fe881c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.139+0000 7fe881c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.139+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c075f10 con 0x7fe87c0b1f60 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.139+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe86c0038a0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.140+0000 7fe863fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:44.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.140+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fe86c0011b0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.140+0000 7fe863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.140+0000 7fe863fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.140+0000 7fe863fff640 10 monclient: dump: 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:44.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.140+0000 7fe8797fa640 4 set_mon_vals no callback set 2024-03-29T10:53:44.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.141+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fe86c003cf0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe87affd640 10 monclient: get_auth_request con 0x7fe86806dfb0 auth_method 0 2024-03-29T10:53:44.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fe86c084b00 con 0x7fe87c0b1f60 2024-03-29T10:53:44.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe863fff640 10 client.4516.objecter ms_dispatch 0x7fe87c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe863fff640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe863fff640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe863fff640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.142+0000 7fe863fff640 10 client.4516.objecter ms_handle_connect 0x7fe86806dfb0 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.147+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe86c0503e0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.147+0000 7fe863fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:44.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.147+0000 7fe863fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:44.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.558+0000 7fe881c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:44.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.558+0000 7fe881c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:44.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.558+0000 7fe881c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:53:44.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.558+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe87c149be0 con 0x7fe87c0b1f60 2024-03-29T10:53:44.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.559+0000 7fe863fff640 1 -- 172.21.15.64:0/97930741 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fe86c052310 con 0x7fe87c0b1f60 2024-03-29T10:53:44.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.559+0000 7fe863fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:44.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.559+0000 7fe863fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:44.540 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:44.540 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.562+0000 7fe881c4e640 10 librados: watch_flush enter 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.562+0000 7fe881c4e640 10 librados: watch_flush exit 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.562+0000 7fe881c4e640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.562+0000 7fe881c4e640 10 client.4516.objecter successfully canceled tick 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 >> v1:172.21.15.174:6816/2473791532 conn(0x7fe86806dfb0 legacy=0x7fe868070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 10 monclient: shutdown 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 >> v1:172.21.15.174:6789/0 conn(0x7fe87c0b1f60 legacy=0x7fe87c070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 shutdown_connections 2024-03-29T10:53:44.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 >> 172.21.15.64:0/97930741 conn(0x7fe87c066a00 msgr2=0x7fe87c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:44.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 shutdown_connections 2024-03-29T10:53:44.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 -- 172.21.15.64:0/97930741 wait complete. 2024-03-29T10:53:44.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:44.563+0000 7fe881c4e640 1 librados: shutdown 2024-03-29T10:53:44.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:51.367 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:51.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.526+0000 7f48984f6640 10 monclient: get_monmap_and_config 2024-03-29T10:53:51.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.526+0000 7f48984f6640 10 monclient: build_initial_monmap 2024-03-29T10:53:51.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.526+0000 7f48984f6640 10 monclient: monmap: 2024-03-29T10:53:51.508 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:51.508 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:51.527665+0000 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:51.527665+0000 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:51.509 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:51.510 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:51.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.527+0000 7f48984f6640 1 Processor -- start 2024-03-29T10:53:51.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.527+0000 7f48984f6640 1 -- start start 2024-03-29T10:53:51.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.527+0000 7f48984f6640 10 monclient: init 2024-03-29T10:53:51.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.528+0000 7f48984f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:51.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.528+0000 7f48984f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:51.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 10 monclient(hunting): picked mon.noname-c con 0x7f48901457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:51.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 10 monclient(hunting): picked mon.noname-a con 0x7f489013f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:51.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 10 monclient(hunting): picked mon.noname-b con 0x7f48900a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48900aaef0 con 0x7f489013f650 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4890145bd0 con 0x7f48901457f0 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4890145eb0 con 0x7f48900a9d70 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f48984f6640 10 monclient(hunting): authenticate will time out at 1242.827922s 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4896a93640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f48900a9d70 0x7f489014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42106/0 (socket says 172.21.15.64:42106) 2024-03-29T10:53:51.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4896a93640 1 -- 172.21.15.64:0/307385324 learned_addr learned my addr 172.21.15.64:0/307385324 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:51.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376998580 0 0) 0x7f4890145eb0 con 0x7f48900a9d70 2024-03-29T10:53:51.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4895290640 10 monclient(hunting): _negotiate 2024-03-29T10:53:51.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4895290640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:51.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4895290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:51.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4895290640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:53:51.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.529+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f487c003520 con 0x7f48900a9d70 2024-03-29T10:53:51.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.530+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146989679 0 0) 0x7f487c003520 con 0x7f48900a9d70 2024-03-29T10:53:51.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.530+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 >> v1:172.21.15.174:6789/0 conn(0x7f48901457f0 legacy=0x7f489013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:51.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 >> v1:172.21.15.64:6789/0 conn(0x7f489013f650 legacy=0x7f48900b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:51.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: _finish_hunting 0 2024-03-29T10:53:51.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 1 monclient: found mon.noname-b 2024-03-29T10:53:51.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:51.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:53:51.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4890146340 con 0x7f48900a9d70 2024-03-29T10:53:51.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: _finish_auth 0 2024-03-29T10:53:51.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: _check_auth_tickets 2024-03-29T10:53:51.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:51.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f48984f6640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:53:51.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f48984f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:51.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4888001a10 con 0x7f48900a9d70 2024-03-29T10:53:51.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:51.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:51.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f4895290640 10 monclient: dump: 2024-03-29T10:53:51.523 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:51.523 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:51.524 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:51.524 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:51.524 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:51.524 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:51.524 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:51.524 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:51.525 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:51.525 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:51.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.533+0000 7f48984f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:51.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.534+0000 7f4895290640 1 -- 172.21.15.64:0/307385324 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4888001be0 con 0x7f48900a9d70 2024-03-29T10:53:51.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.534+0000 7f4895290640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:51.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.534+0000 7f48984f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:51.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.534+0000 7f48984f6640 4 set_mon_vals no callback set 2024-03-29T10:53:51.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.535+0000 7f48984f6640 10 monclient: shutdown 2024-03-29T10:53:51.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.535+0000 7f48984f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:51.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.535+0000 7f48984f6640 1 -- 172.21.15.64:0/307385324 >> v1:172.21.15.64:6790/0 conn(0x7f48900a9d70 legacy=0x7f489014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:51.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.535+0000 7f48984f6640 1 -- 172.21.15.64:0/307385324 shutdown_connections 2024-03-29T10:53:51.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.535+0000 7f48984f6640 1 -- 172.21.15.64:0/307385324 >> 172.21.15.64:0/307385324 conn(0x7f4890066a00 msgr2=0x7f4890066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:51.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.536+0000 7f48984f6640 1 -- 172.21.15.64:0/307385324 shutdown_connections 2024-03-29T10:53:51.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.536+0000 7f48984f6640 1 -- 172.21.15.64:0/307385324 wait complete. 2024-03-29T10:53:51.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 10 monclient: build_initial_monmap 2024-03-29T10:53:51.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 10 monclient: monmap: 2024-03-29T10:53:51.527 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:51.527 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:51.528 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:51.528 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:51.528 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:51.528 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:51.528 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:51.528 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:51.529 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:51.529 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:51.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 1 librados: starting msgr at 2024-03-29T10:53:51.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 1 librados: starting objecter 2024-03-29T10:53:51.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 1 Processor -- start 2024-03-29T10:53:51.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 1 -- start start 2024-03-29T10:53:51.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 1 librados: setting wanted keys 2024-03-29T10:53:51.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 1 librados: calling monclient init 2024-03-29T10:53:51.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.537+0000 7f48984f6640 10 monclient: init 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient(hunting): picked mon.noname-c con 0x7f48900a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient(hunting): picked mon.noname-b con 0x7f489013f650 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient(hunting): picked mon.noname-a con 0x7f48901457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489006e660 con 0x7f48901457f0 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489006e940 con 0x7f489013f650 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48901dc1f0 con 0x7f48900a9d70 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.538+0000 7f48984f6640 10 monclient(hunting): authenticate will time out at 1242.837499s 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.541+0000 7f4896292640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f48900a9d70 0x7f489013aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42938/0 (socket says 172.21.15.64:42938) 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.542+0000 7f4896292640 1 -- 172.21.15.64:0/2067061406 learned_addr learned my addr 172.21.15.64:0/2067061406 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:51.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.542+0000 7f4886ffd640 10 client.?.objecter ms_handle_connect 0x7f48900a9d70 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.542+0000 7f4886ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.542+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380560396 0 0) 0x7f48901dc1f0 con 0x7f48900a9d70 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.542+0000 7f4886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4874004eb0 con 0x7f48900a9d70 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 10 client.?.objecter ms_handle_connect 0x7f48901457f0 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210876291 0 0) 0x7f4874004eb0 con 0x7f48900a9d70 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.543+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48901dc1f0 con 0x7f48900a9d70 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f488c003fe0 con 0x7f48900a9d70 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:51.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 client.?.objecter ms_handle_connect 0x7f489013f650 2024-03-29T10:53:51.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262899459 0 0) 0x7f489006e660 con 0x7f48901457f0 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4874004eb0 con 0x7f48901457f0 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1765108703 0 0) 0x7f48901dc1f0 con 0x7f48900a9d70 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 >> v1:172.21.15.174:6789/0 conn(0x7f489013f650 legacy=0x7f48901db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 >> v1:172.21.15.64:6789/0 conn(0x7f48901457f0 legacy=0x7f48901dbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 monclient: found mon.c 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:51.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4890138a90 con 0x7f48900a9d70 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.544+0000 7f4886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 5 monclient: authenticate success, global_id 4541 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 10 monclient: _renew_subs 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489013b5c0 con 0x7f48900a9d70 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 10 monclient: _renew_subs 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f489013a560 con 0x7f48900a9d70 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.545+0000 7f48984f6640 1 librados: init done 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.547+0000 7f48984f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:51.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.547+0000 7f48984f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.547+0000 7f48984f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.547+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489006ee60 con 0x7f48900a9d70 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f488c004140 con 0x7f48900a9d70 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f488c000ed0 con 0x7f48900a9d70 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4886ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4886ffd640 10 monclient: dump: 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:51.547 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.553+0000 7f4894a8f640 4 set_mon_vals no callback set 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.554+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f488c0010f0 con 0x7f48900a9d70 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.555+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f488c0847d0 con 0x7f48900a9d70 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.555+0000 7f4886ffd640 10 client.4541.objecter ms_dispatch 0x7f4890000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.555+0000 7f4886ffd640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.555+0000 7f4886ffd640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.555+0000 7f4886ffd640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.555+0000 7f4895a91640 10 monclient: get_auth_request con 0x7f487406dcc0 auth_method 0 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.556+0000 7f4886ffd640 10 client.4541.objecter ms_handle_connect 0x7f487406dcc0 2024-03-29T10:53:51.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.558+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f488c0500b0 con 0x7f48900a9d70 2024-03-29T10:53:51.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.558+0000 7f4886ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:51.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.558+0000 7f4886ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:51.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.893+0000 7f48984f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:51.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.893+0000 7f48984f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:51.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.893+0000 7f48984f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:51.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.893+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f489013aaf0 con 0x7f48900a9d70 2024-03-29T10:53:51.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.894+0000 7f4886ffd640 1 -- 172.21.15.64:0/2067061406 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f488c051fe0 con 0x7f48900a9d70 2024-03-29T10:53:51.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.895+0000 7f4886ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:51.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.895+0000 7f4886ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:51.876 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:51.876 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:51.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 10 librados: watch_flush enter 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 10 librados: watch_flush exit 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 10 client.4541.objecter successfully canceled tick 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 >> v1:172.21.15.174:6816/2473791532 conn(0x7f487406dcc0 legacy=0x7f4874070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 10 monclient: shutdown 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 >> v1:172.21.15.64:6790/0 conn(0x7f48900a9d70 legacy=0x7f489013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 shutdown_connections 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 >> 172.21.15.64:0/2067061406 conn(0x7f4890066a00 msgr2=0x7f48900b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.898+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 shutdown_connections 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.899+0000 7f48984f6640 1 -- 172.21.15.64:0/2067061406 wait complete. 2024-03-29T10:53:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:51.899+0000 7f48984f6640 1 librados: shutdown 2024-03-29T10:53:51.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:58.700 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.849+0000 7fb2963ab640 10 monclient: get_monmap_and_config 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.849+0000 7fb2963ab640 10 monclient: build_initial_monmap 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.849+0000 7fb2963ab640 10 monclient: monmap: 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:53:58.851640+0000 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:53:58.851640+0000 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:58.833 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:53:58.834 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:53:58.834 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:58.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.850+0000 7fb2963ab640 1 Processor -- start 2024-03-29T10:53:58.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.850+0000 7fb2963ab640 1 -- start start 2024-03-29T10:53:58.834 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.850+0000 7fb2963ab640 10 monclient: init 2024-03-29T10:53:58.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:58.836 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient(hunting): picked mon.noname-a con 0x7fb2880350e0 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient(hunting): picked mon.noname-c con 0x7fb2880c14f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient(hunting): picked mon.noname-b con 0x7fb2880c4670 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28802b560 con 0x7fb2880350e0 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28802c430 con 0x7fb2880c14f0 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28802c6a0 con 0x7fb2880c4670 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.851+0000 7fb2963ab640 10 monclient(hunting): authenticate will time out at 1250.151711s 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.852+0000 7fb295baa640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb2880c4670 0x7fb2880cf5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35070/0 (socket says 172.21.15.64:35070) 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.852+0000 7fb295baa640 1 -- 172.21.15.64:0/97567221 learned_addr learned my addr 172.21.15.64:0/97567221 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992484852 0 0) 0x7fb28802c6a0 con 0x7fb2880c4670 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:58.837 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27c003520 con 0x7fb2880c4670 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868461076 0 0) 0x7fb28802c430 con 0x7fb2880c14f0 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.853+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb28802c6a0 con 0x7fb2880c14f0 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.854+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 847653756 0 0) 0x7fb27c003520 con 0x7fb2880c4670 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.854+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 >> v1:172.21.15.174:6789/0 conn(0x7fb2880c14f0 legacy=0x7fb2880c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:58.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.854+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 >> v1:172.21.15.64:6789/0 conn(0x7fb2880350e0 legacy=0x7fb2880c0de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:58.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:58.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 1 monclient: found mon.noname-b 2024-03-29T10:53:58.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:58.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:53:58.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28802c980 con 0x7fb2880c4670 2024-03-29T10:53:58.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb2963ab640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb2963ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb29004f730 con 0x7fb2880c4670 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb283fff640 10 monclient: dump: 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:58.845 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.855+0000 7fb2963ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb290075f10 con 0x7fb2880c4670 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb283fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb2963ab640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb2963ab640 4 set_mon_vals no callback set 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb283fff640 1 -- 172.21.15.64:0/97567221 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb2900758b0 con 0x7fb2880c4670 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb283fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:58.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.856+0000 7fb283fff640 10 monclient: dump: 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.857+0000 7fb2963ab640 10 monclient: shutdown 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.857+0000 7fb2963ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.857+0000 7fb2963ab640 1 -- 172.21.15.64:0/97567221 >> v1:172.21.15.64:6790/0 conn(0x7fb2880c4670 legacy=0x7fb2880cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.857+0000 7fb2963ab640 1 -- 172.21.15.64:0/97567221 shutdown_connections 2024-03-29T10:53:58.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.857+0000 7fb2963ab640 1 -- 172.21.15.64:0/97567221 >> 172.21.15.64:0/97567221 conn(0x7fb2880058f0 msgr2=0x7fb288005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.858+0000 7fb2963ab640 1 -- 172.21.15.64:0/97567221 shutdown_connections 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.858+0000 7fb2963ab640 1 -- 172.21.15.64:0/97567221 wait complete. 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.858+0000 7fb2963ab640 10 monclient: build_initial_monmap 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.858+0000 7fb2963ab640 10 monclient: monmap: 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:53:58.848 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 librados: starting msgr at 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 librados: starting objecter 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 Processor -- start 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 -- start start 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 librados: setting wanted keys 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 librados: calling monclient init 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient: init 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient(hunting): picked mon.noname-b con 0x7fb2880350e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient(hunting): picked mon.noname-c con 0x7fb2880c14f0 addr v1:172.21.15.64:6790/0 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient(hunting): picked mon.noname-a con 0x7fb2880c4670 addr v1:172.21.15.64:6789/0 2024-03-29T10:53:58.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2880c4c10 con 0x7fb2880c4670 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2880b7b40 con 0x7fb2880350e0 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2880b7d70 con 0x7fb2880c14f0 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.859+0000 7fb2963ab640 10 monclient(hunting): authenticate will time out at 1250.159832s 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb294ba8640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb2880c14f0 0x7fb28802e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36878/0 (socket says 172.21.15.64:36878) 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb294ba8640 1 -- 172.21.15.64:0/3514378091 learned_addr learned my addr 172.21.15.64:0/3514378091 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 10 client.?.objecter ms_handle_connect 0x7fb2880c14f0 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536403461 0 0) 0x7fb2880b7d70 con 0x7fb2880c14f0 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:53:58.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.860+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb274004eb0 con 0x7fb2880c14f0 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 10 client.?.objecter ms_handle_connect 0x7fb2880350e0 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378415645 0 0) 0x7fb274004eb0 con 0x7fb2880c14f0 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2880b7d70 con 0x7fb2880c14f0 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb28c001a10 con 0x7fb2880c14f0 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:58.853 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 615802703 0 0) 0x7fb2880b7d70 con 0x7fb2880c14f0 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 >> v1:172.21.15.174:6789/0 conn(0x7fb2880350e0 legacy=0x7fb2880319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.861+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 >> v1:172.21.15.64:6789/0 conn(0x7fb2880c4670 legacy=0x7fb28802ec10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:58.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 1 monclient: found mon.c 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2880b8050 con 0x7fb2880c14f0 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb2963ab640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb2963ab640 10 monclient: _renew_subs 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb2963ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.863+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2880b8330 con 0x7fb2880c14f0 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb2963ab640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb2963ab640 10 monclient: _renew_subs 2024-03-29T10:53:58.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb2963ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb288165fa0 con 0x7fb2880c14f0 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb2963ab640 1 librados: init done 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb28c001cd0 con 0x7fb2880c14f0 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb281ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.864+0000 7fb2837fe640 4 set_mon_vals no callback set 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fb28c003a60 con 0x7fb2880c14f0 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb281ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb281ffb640 10 monclient: dump: 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:53:58.856 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb2963ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb2963ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb2963ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.865+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb270005080 con 0x7fb2880c14f0 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.866+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fb28c003c40 con 0x7fb2880c14f0 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.866+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fb28c084bf0 con 0x7fb2880c14f0 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.866+0000 7fb281ffb640 10 client.4547.objecter ms_dispatch 0x7fb290000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.866+0000 7fb281ffb640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:58.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.866+0000 7fb281ffb640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:58.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.866+0000 7fb281ffb640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T10:53:58.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.867+0000 7fb2953a9640 10 monclient: get_auth_request con 0x7fb27406dc40 auth_method 0 2024-03-29T10:53:58.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.868+0000 7fb281ffb640 10 client.4547.objecter ms_handle_connect 0x7fb27406dc40 2024-03-29T10:53:58.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.874+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb28c050530 con 0x7fb2880c14f0 2024-03-29T10:53:58.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.874+0000 7fb281ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:58.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:58.874+0000 7fb281ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:59.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.191+0000 7fb2963ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:59.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.191+0000 7fb2963ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:59.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.191+0000 7fb2963ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:53:59.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.191+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb270005330 con 0x7fb2880c14f0 2024-03-29T10:53:59.175 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:53:59.175 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:59.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.192+0000 7fb281ffb640 1 -- 172.21.15.64:0/3514378091 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fb28c0842a0 con 0x7fb2880c14f0 2024-03-29T10:53:59.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.192+0000 7fb281ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:59.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.192+0000 7fb281ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 10 librados: watch_flush enter 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 10 librados: watch_flush exit 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 10 client.4547.objecter successfully canceled tick 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 >> v1:172.21.15.174:6816/2473791532 conn(0x7fb27406dc40 legacy=0x7fb274070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 10 monclient: shutdown 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:59.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 >> v1:172.21.15.64:6790/0 conn(0x7fb2880c14f0 legacy=0x7fb28802e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:59.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 shutdown_connections 2024-03-29T10:53:59.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 >> 172.21.15.64:0/3514378091 conn(0x7fb2880058f0 msgr2=0x7fb2880c0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:59.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.196+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 shutdown_connections 2024-03-29T10:53:59.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.197+0000 7fb2963ab640 1 -- 172.21.15.64:0/3514378091 wait complete. 2024-03-29T10:53:59.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:53:59.197+0000 7fb2963ab640 1 librados: shutdown 2024-03-29T10:53:59.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:06.000 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.170+0000 7f47f58ba640 10 monclient: get_monmap_and_config 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.170+0000 7f47f58ba640 10 monclient: build_initial_monmap 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.170+0000 7f47f58ba640 10 monclient: monmap: 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:06.171103+0000 2024-03-29T10:54:06.151 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:06.171103+0000 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.171+0000 7f47f58ba640 1 Processor -- start 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.171+0000 7f47f58ba640 1 -- start start 2024-03-29T10:54:06.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.171+0000 7f47f58ba640 10 monclient: init 2024-03-29T10:54:06.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:06.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient(hunting): picked mon.noname-c con 0x7f47f01457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient(hunting): picked mon.noname-b con 0x7f47f013f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient(hunting): picked mon.noname-a con 0x7f47f00a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00aaef0 con 0x7f47f00a9d70 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0145bd0 con 0x7f47f01457f0 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0145eb0 con 0x7f47f013f650 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.172+0000 7f47f58ba640 10 monclient(hunting): authenticate will time out at 1257.470969s 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.174+0000 7f47ef7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f47f00a9d70 0x7f47f014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40849/0 (socket says 172.21.15.64:40849) 2024-03-29T10:54:06.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.174+0000 7f47ef7fe640 1 -- 172.21.15.64:0/3582366151 learned_addr learned my addr 172.21.15.64:0/3582366151 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:06.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539764826 0 0) 0x7f47f0145bd0 con 0x7f47f01457f0 2024-03-29T10:54:06.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:06.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:06.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47dc003520 con 0x7f47f01457f0 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478260056 0 0) 0x7f47f00aaef0 con 0x7f47f00a9d70 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.175+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f0145bd0 con 0x7f47f00a9d70 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.176+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135820292 0 0) 0x7f47f0145bd0 con 0x7f47f00a9d70 2024-03-29T10:54:06.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.176+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 >> v1:172.21.15.64:6790/0 conn(0x7f47f013f650 legacy=0x7f47f00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 >> v1:172.21.15.174:6789/0 conn(0x7f47f01457f0 legacy=0x7f47f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 1 monclient: found mon.noname-a 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0146340 con 0x7f47f00a9d70 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:06.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.177+0000 7f47edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47f58ba640 5 monclient: authenticate success, global_id 4494 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47f58ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f47e80015f0 con 0x7f47f00a9d70 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47edffb640 10 monclient: dump: 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:06.162 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47f58ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47e8001810 con 0x7f47f00a9d70 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47f58ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.178+0000 7f47f58ba640 4 set_mon_vals no callback set 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47edffb640 1 -- 172.21.15.64:0/3582366151 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f47e8001ba0 con 0x7f47f00a9d70 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47edffb640 10 monclient: dump: 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:06.163 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47f58ba640 10 monclient: shutdown 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47f58ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47f58ba640 1 -- 172.21.15.64:0/3582366151 >> v1:172.21.15.64:6789/0 conn(0x7f47f00a9d70 legacy=0x7f47f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47f58ba640 1 -- 172.21.15.64:0/3582366151 shutdown_connections 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.179+0000 7f47f58ba640 1 -- 172.21.15.64:0/3582366151 >> 172.21.15.64:0/3582366151 conn(0x7f47f0066a00 msgr2=0x7f47f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:06.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.180+0000 7f47f58ba640 1 -- 172.21.15.64:0/3582366151 shutdown_connections 2024-03-29T10:54:06.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.180+0000 7f47f58ba640 1 -- 172.21.15.64:0/3582366151 wait complete. 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.181+0000 7f47f58ba640 10 monclient: build_initial_monmap 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.181+0000 7f47f58ba640 10 monclient: monmap: 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:06.171 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.181+0000 7f47f58ba640 1 librados: starting msgr at 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.181+0000 7f47f58ba640 1 librados: starting objecter 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.181+0000 7f47f58ba640 1 Processor -- start 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 1 -- start start 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 1 librados: setting wanted keys 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 1 librados: calling monclient init 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient: init 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:06.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient(hunting): picked mon.noname-c con 0x7f47f00a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient(hunting): picked mon.noname-a con 0x7f47f013f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient(hunting): picked mon.noname-b con 0x7f47f01457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f006e8b0 con 0x7f47f013f650 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0070c30 con 0x7f47f01457f0 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0071db0 con 0x7f47f00a9d70 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.182+0000 7f47f58ba640 10 monclient(hunting): authenticate will time out at 1257.481778s 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47ee7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f47f013f650 0x7f47f01dbb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40933/0 (socket says 172.21.15.64:40933) 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47ee7fc640 1 -- 172.21.15.64:0/1291929132 learned_addr learned my addr 172.21.15.64:0/1291929132 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 client.?.objecter ms_handle_connect 0x7f47f013f650 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868055683 0 0) 0x7f47f006e8b0 con 0x7f47f013f650 2024-03-29T10:54:06.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d0004eb0 con 0x7f47f013f650 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 client.?.objecter ms_handle_connect 0x7f47f01457f0 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291958965 0 0) 0x7f47d0004eb0 con 0x7f47f013f650 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f006e8b0 con 0x7f47f013f650 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f47d8001960 con 0x7f47f013f650 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.186+0000 7f47e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:06.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient(hunting): dump: 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1325939363 0 0) 0x7f47f006e8b0 con 0x7f47f013f650 2024-03-29T10:54:06.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 >> v1:172.21.15.64:6790/0 conn(0x7f47f00a9d70 legacy=0x7f47f00716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 >> v1:172.21.15.174:6789/0 conn(0x7f47f01457f0 legacy=0x7f47f01e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 1 monclient: found mon.a 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0139340 con 0x7f47f013f650 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 client.?.objecter ms_handle_connect 0x7f47f00a9d70 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:54:06.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 10 monclient: _renew_subs 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f0139590 con 0x7f47f013f650 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 10 monclient: _renew_subs 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f01dc400 con 0x7f47f013f650 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.187+0000 7f47f58ba640 1 librados: init done 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d8001c40 con 0x7f47f013f650 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f47d8003a40 con 0x7f47f013f650 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47e7fff640 10 monclient: dump: 2024-03-29T10:54:06.177 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.189+0000 7f47ed7fa640 4 set_mon_vals no callback set 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.190+0000 7f47f58ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:06.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.190+0000 7f47f58ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:06.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.190+0000 7f47f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:06.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.190+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d4005080 con 0x7f47f013f650 2024-03-29T10:54:06.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.191+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f47d8003ba0 con 0x7f47f013f650 2024-03-29T10:54:06.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.192+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f47d8084ad0 con 0x7f47f013f650 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.192+0000 7f47e7fff640 10 client.4497.objecter ms_dispatch 0x7f47f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.192+0000 7f47e7fff640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.192+0000 7f47e7fff640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.192+0000 7f47e7fff640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.193+0000 7f47eeffd640 10 monclient: get_auth_request con 0x7f47d006dd60 auth_method 0 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.193+0000 7f47e7fff640 10 client.4497.objecter ms_handle_connect 0x7f47d006dd60 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.198+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47d8086e60 con 0x7f47f013f650 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.198+0000 7f47e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:06.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.198+0000 7f47e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:06.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.529+0000 7f47f58ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:06.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.529+0000 7f47f58ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:06.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.529+0000 7f47f58ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:06.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.529+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d4005330 con 0x7f47f013f650 2024-03-29T10:54:06.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.530+0000 7f47e7fff640 1 -- 172.21.15.64:0/1291929132 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f47d80503b0 con 0x7f47f013f650 2024-03-29T10:54:06.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.530+0000 7f47e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:06.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.530+0000 7f47e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:06.511 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:06.511 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:06.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 10 librados: watch_flush enter 2024-03-29T10:54:06.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 10 librados: watch_flush exit 2024-03-29T10:54:06.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-29T10:54:06.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 10 client.4497.objecter successfully canceled tick 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 >> v1:172.21.15.174:6816/2473791532 conn(0x7f47d006dd60 legacy=0x7f47d0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 10 monclient: shutdown 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.533+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 >> v1:172.21.15.64:6789/0 conn(0x7f47f013f650 legacy=0x7f47f01dbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.534+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 shutdown_connections 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.534+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 >> 172.21.15.64:0/1291929132 conn(0x7f47f0066a00 msgr2=0x7f47f013c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.534+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 shutdown_connections 2024-03-29T10:54:06.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.534+0000 7f47f58ba640 1 -- 172.21.15.64:0/1291929132 wait complete. 2024-03-29T10:54:06.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:06.534+0000 7f47f58ba640 1 librados: shutdown 2024-03-29T10:54:06.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:13.340 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.515+0000 7f606d779640 10 monclient: get_monmap_and_config 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.515+0000 7f606d779640 10 monclient: build_initial_monmap 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.515+0000 7f606d779640 10 monclient: monmap: 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:13.516580+0000 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:13.516580+0000 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:13.498 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:13.499 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:13.499 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:13.499 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:13.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.517+0000 7f606d779640 1 Processor -- start 2024-03-29T10:54:13.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.517+0000 7f606d779640 1 -- start start 2024-03-29T10:54:13.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.517+0000 7f606d779640 10 monclient: init 2024-03-29T10:54:13.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:13.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:13.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient(hunting): picked mon.noname-a con 0x7f60680a9ac0 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:13.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient(hunting): picked mon.noname-c con 0x7f606813ee90 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:13.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient(hunting): picked mon.noname-b con 0x7f60680af900 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:13.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680afd10 con 0x7f60680a9ac0 2024-03-29T10:54:13.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680af6f0 con 0x7f606813ee90 2024-03-29T10:54:13.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680aac80 con 0x7f60680af900 2024-03-29T10:54:13.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:13.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.518+0000 7f606d779640 10 monclient(hunting): authenticate will time out at 1264.817006s 2024-03-29T10:54:13.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.522+0000 7f6066ffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f60680a9ac0 0x7f60680adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51053/0 (socket says 172.21.15.64:51053) 2024-03-29T10:54:13.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.522+0000 7f6066ffd640 1 -- 172.21.15.64:0/2441497783 learned_addr learned my addr 172.21.15.64:0/2441497783 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:13.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952845336 0 0) 0x7f60680afd10 con 0x7f60680a9ac0 2024-03-29T10:54:13.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:13.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:13.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:13.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:54:13.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6050003520 con 0x7f60680a9ac0 2024-03-29T10:54:13.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387214847 0 0) 0x7f60680af6f0 con 0x7f606813ee90 2024-03-29T10:54:13.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:13.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:13.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:13.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:54:13.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60680afd10 con 0x7f606813ee90 2024-03-29T10:54:13.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268117401 0 0) 0x7f60680aac80 con 0x7f60680af900 2024-03-29T10:54:13.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:13.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:13.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:13.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:54:13.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60680af6f0 con 0x7f60680af900 2024-03-29T10:54:13.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474222151 0 0) 0x7f6050003520 con 0x7f60680a9ac0 2024-03-29T10:54:13.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 >> v1:172.21.15.64:6790/0 conn(0x7f60680af900 legacy=0x7f6068149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 >> v1:172.21.15.174:6789/0 conn(0x7f606813ee90 legacy=0x7f606813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:13.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.526+0000 7f6065ffb640 1 monclient: found mon.noname-a 2024-03-29T10:54:13.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:13.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:54:13.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60680aaf60 con 0x7f60680a9ac0 2024-03-29T10:54:13.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6060001a10 con 0x7f60680a9ac0 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: dump: 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:13.516 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f606d779640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:54:13.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f606d779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:13.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6060001c30 con 0x7f60680a9ac0 2024-03-29T10:54:13.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:13.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 1 -- 172.21.15.64:0/2441497783 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6060001fc0 con 0x7f60680a9ac0 2024-03-29T10:54:13.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:13.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:13.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.527+0000 7f6065ffb640 10 monclient: dump: 2024-03-29T10:54:13.519 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:13.519 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:13.519 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.519 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.519 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:13.519 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.528+0000 7f606d779640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.528+0000 7f606d779640 4 set_mon_vals no callback set 2024-03-29T10:54:13.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.529+0000 7f606d779640 10 monclient: shutdown 2024-03-29T10:54:13.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.529+0000 7f606d779640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:13.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.529+0000 7f606d779640 1 -- 172.21.15.64:0/2441497783 >> v1:172.21.15.64:6789/0 conn(0x7f60680a9ac0 legacy=0x7f60680adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.532+0000 7f606d779640 1 -- 172.21.15.64:0/2441497783 shutdown_connections 2024-03-29T10:54:13.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.532+0000 7f606d779640 1 -- 172.21.15.64:0/2441497783 >> 172.21.15.64:0/2441497783 conn(0x7f60680b35e0 msgr2=0x7f60680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:13.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.533+0000 7f606d779640 1 -- 172.21.15.64:0/2441497783 shutdown_connections 2024-03-29T10:54:13.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.533+0000 7f606d779640 1 -- 172.21.15.64:0/2441497783 wait complete. 2024-03-29T10:54:13.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.534+0000 7f606d779640 10 monclient: build_initial_monmap 2024-03-29T10:54:13.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.534+0000 7f606d779640 10 monclient: monmap: 2024-03-29T10:54:13.530 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:13.530 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:13.530 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:13.530 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.534+0000 7f606d779640 1 librados: starting msgr at 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.534+0000 7f606d779640 1 librados: starting objecter 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.534+0000 7f606d779640 1 Processor -- start 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.535+0000 7f606d779640 1 -- start start 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.535+0000 7f606d779640 1 librados: setting wanted keys 2024-03-29T10:54:13.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.535+0000 7f606d779640 1 librados: calling monclient init 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.535+0000 7f606d779640 10 monclient: init 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.535+0000 7f606d779640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.535+0000 7f606d779640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.538+0000 7f606d779640 10 monclient(hunting): picked mon.noname-c con 0x7f60680a9ac0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.538+0000 7f6066ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f60680a9ac0 0x7f60681defd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59627/0 (socket says 172.21.15.64:59627) 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.539+0000 7f606d779640 10 monclient(hunting): picked mon.noname-a con 0x7f60680af900 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f606d779640 10 monclient(hunting): picked mon.noname-b con 0x7f606813ee90 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f6066ffd640 1 -- 172.21.15.64:0/3067771425 learned_addr learned my addr 172.21.15.64:0/3067771425 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:13.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680716c0 con 0x7f60680af900 2024-03-29T10:54:13.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606811b680 con 0x7f606813ee90 2024-03-29T10:54:13.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6068071b50 con 0x7f60680a9ac0 2024-03-29T10:54:13.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f606d779640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:13.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f606d779640 10 monclient(hunting): authenticate will time out at 1264.841594s 2024-03-29T10:54:13.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f604f7fe640 10 client.?.objecter ms_handle_connect 0x7f60680a9ac0 2024-03-29T10:54:13.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f604f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:13.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.542+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982186110 0 0) 0x7f6068071b50 con 0x7f60680a9ac0 2024-03-29T10:54:13.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:13.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:13.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:13.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:54:13.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6054004eb0 con 0x7f60680a9ac0 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 client.?.objecter ms_handle_connect 0x7f606813ee90 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500173613 0 0) 0x7f6054004eb0 con 0x7f60680a9ac0 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6068071b50 con 0x7f60680a9ac0 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6060001a10 con 0x7f60680a9ac0 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): dump: 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:13.535 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3151849632 0 0) 0x7f6068071b50 con 0x7f60680a9ac0 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 >> v1:172.21.15.174:6789/0 conn(0x7f606813ee90 legacy=0x7f60681e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 >> v1:172.21.15.64:6789/0 conn(0x7f60680af900 legacy=0x7f6068138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 monclient: found mon.c 2024-03-29T10:54:13.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6068071ce0 con 0x7f60680a9ac0 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6060004e80 con 0x7f60680a9ac0 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.543+0000 7f604f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f6060005250 con 0x7f60680a9ac0 2024-03-29T10:54:13.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f604f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f604f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f604f7fe640 10 monclient: dump: 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:13.538 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 10 monclient: _renew_subs 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60680711e0 con 0x7f60680a9ac0 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 10 monclient: _renew_subs 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60680708b0 con 0x7f60680a9ac0 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f606d779640 1 librados: init done 2024-03-29T10:54:13.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.544+0000 7f60657fa640 4 set_mon_vals no callback set 2024-03-29T10:54:13.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.547+0000 7f606d779640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:13.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.547+0000 7f606d779640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:13.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.547+0000 7f606d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:13.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.547+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048005080 con 0x7f60680a9ac0 2024-03-29T10:54:13.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.548+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f60600053b0 con 0x7f60680a9ac0 2024-03-29T10:54:13.540 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.549+0000 7f60667fc640 10 monclient: get_auth_request con 0x7f605406dc20 auth_method 0 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.549+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f6060084fa0 con 0x7f60680a9ac0 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.549+0000 7f604f7fe640 10 client.4556.objecter ms_dispatch 0x7f6068000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.549+0000 7f604f7fe640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.549+0000 7f604f7fe640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.549+0000 7f604f7fe640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.551+0000 7f604f7fe640 10 client.4556.objecter ms_handle_connect 0x7f605406dc20 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.557+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60600506b0 con 0x7f60680a9ac0 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.557+0000 7f604f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:13.541 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.557+0000 7f604f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:13.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.888+0000 7f606d779640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:13.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.888+0000 7f606d779640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:13.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.888+0000 7f606d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:13.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.888+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6048005330 con 0x7f60680a9ac0 2024-03-29T10:54:13.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.889+0000 7f604f7fe640 1 -- 172.21.15.64:0/3067771425 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f60600527b0 con 0x7f60680a9ac0 2024-03-29T10:54:13.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.889+0000 7f604f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:13.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.889+0000 7f604f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:13.874 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:13.875 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 10 librados: watch_flush enter 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 10 librados: watch_flush exit 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 10 client.4556.objecter successfully canceled tick 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 >> v1:172.21.15.174:6816/2473791532 conn(0x7f605406dc20 legacy=0x7f60540700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 10 monclient: shutdown 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.893+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 >> v1:172.21.15.64:6790/0 conn(0x7f60680a9ac0 legacy=0x7f60681defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.894+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 shutdown_connections 2024-03-29T10:54:13.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.894+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 >> 172.21.15.64:0/3067771425 conn(0x7f60680b35e0 msgr2=0x7f60680ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:13.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.895+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 shutdown_connections 2024-03-29T10:54:13.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.898+0000 7f606d779640 1 -- 172.21.15.64:0/3067771425 wait complete. 2024-03-29T10:54:13.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:13.898+0000 7f606d779640 1 librados: shutdown 2024-03-29T10:54:13.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:20.704 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:20.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.868+0000 7fce4ac99640 10 monclient: get_monmap_and_config 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.868+0000 7fce4ac99640 10 monclient: build_initial_monmap 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.868+0000 7fce4ac99640 10 monclient: monmap: 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:20.869667+0000 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:20.869667+0000 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:20.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 1 Processor -- start 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 1 -- start start 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient: init 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient(hunting): picked mon.noname-a con 0x7fce440a9750 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient(hunting): picked mon.noname-c con 0x7fce440ae930 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient(hunting): picked mon.noname-b con 0x7fce440acab0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce440ad650 con 0x7fce440a9750 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce440aedc0 con 0x7fce440ae930 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce440ab750 con 0x7fce440acab0 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:20.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.870+0000 7fce4ac99640 10 monclient(hunting): authenticate will time out at 1272.169817s 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce49c97640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fce440a9750 0x7fce440ae220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58199/0 (socket says 172.21.15.64:58199) 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce49c97640 1 -- 172.21.15.64:0/1616338962 learned_addr learned my addr 172.21.15.64:0/1616338962 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262430401 0 0) 0x7fce40004a50 con 0x7fce440a9750 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce48c95640 10 monclient(hunting): _negotiate 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce48c95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce48c95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce48c95640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.874+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce30003500 con 0x7fce440a9750 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.875+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897303784 0 0) 0x7fce30003500 con 0x7fce440a9750 2024-03-29T10:54:20.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.875+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 >> v1:172.21.15.64:6790/0 conn(0x7fce440acab0 legacy=0x7fce440acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:20.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.875+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 >> v1:172.21.15.174:6789/0 conn(0x7fce440ae930 legacy=0x7fce440ac3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:20.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: _finish_hunting 0 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 1 monclient: found mon.noname-a 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce440ab9c0 con 0x7fce440a9750 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: _finish_auth 0 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: _check_auth_tickets 2024-03-29T10:54:20.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fce40001290 con 0x7fce440a9750 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce48c95640 10 monclient: dump: 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:20.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce4ac99640 5 monclient: authenticate success, global_id 4503 2024-03-29T10:54:20.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.878+0000 7fce4ac99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce40001460 con 0x7fce440a9750 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce48c95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce48c95640 1 -- 172.21.15.64:0/1616338962 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fce400018b0 con 0x7fce440a9750 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce48c95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce48c95640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce48c95640 10 monclient: dump: 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:20.864 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce4ac99640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.879+0000 7fce4ac99640 4 set_mon_vals no callback set 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.880+0000 7fce4ac99640 10 monclient: shutdown 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.880+0000 7fce4ac99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.880+0000 7fce4ac99640 1 -- 172.21.15.64:0/1616338962 >> v1:172.21.15.64:6789/0 conn(0x7fce440a9750 legacy=0x7fce440ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.881+0000 7fce4ac99640 1 -- 172.21.15.64:0/1616338962 shutdown_connections 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.881+0000 7fce4ac99640 1 -- 172.21.15.64:0/1616338962 >> 172.21.15.64:0/1616338962 conn(0x7fce440673e0 msgr2=0x7fce440b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:20.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.881+0000 7fce4ac99640 1 -- 172.21.15.64:0/1616338962 shutdown_connections 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.881+0000 7fce4ac99640 1 -- 172.21.15.64:0/1616338962 wait complete. 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.882+0000 7fce4ac99640 10 monclient: build_initial_monmap 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.882+0000 7fce4ac99640 10 monclient: monmap: 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:20.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.882+0000 7fce4ac99640 1 librados: starting msgr at 2024-03-29T10:54:20.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.882+0000 7fce4ac99640 1 librados: starting objecter 2024-03-29T10:54:20.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.882+0000 7fce4ac99640 1 Processor -- start 2024-03-29T10:54:20.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.883+0000 7fce4ac99640 1 -- start start 2024-03-29T10:54:20.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.883+0000 7fce4ac99640 1 librados: setting wanted keys 2024-03-29T10:54:20.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.883+0000 7fce4ac99640 1 librados: calling monclient init 2024-03-29T10:54:20.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.883+0000 7fce4ac99640 10 monclient: init 2024-03-29T10:54:20.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.883+0000 7fce4ac99640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.883+0000 7fce4ac99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.884+0000 7fce4ac99640 10 monclient(hunting): picked mon.noname-a con 0x7fce440a9750 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 10 monclient(hunting): picked mon.noname-b con 0x7fce440acab0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 10 monclient(hunting): picked mon.noname-c con 0x7fce440ae930 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce44088e00 con 0x7fce440a9750 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce440890e0 con 0x7fce440acab0 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce440893c0 con 0x7fce440ae930 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:20.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.886+0000 7fce4ac99640 10 monclient(hunting): authenticate will time out at 1272.185676s 2024-03-29T10:54:20.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.890+0000 7fce49c97640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fce440a9750 0x7fce4414dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58379/0 (socket says 172.21.15.64:58379) 2024-03-29T10:54:20.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.890+0000 7fce49c97640 1 -- 172.21.15.64:0/1389819046 learned_addr learned my addr 172.21.15.64:0/1389819046 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:20.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.891+0000 7fce3a7fc640 10 client.?.objecter ms_handle_connect 0x7fce440a9750 2024-03-29T10:54:20.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.891+0000 7fce3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 client.?.objecter ms_handle_connect 0x7fce440acab0 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896360129 0 0) 0x7fce44088e00 con 0x7fce440a9750 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce28004eb0 con 0x7fce440a9750 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 client.?.objecter ms_handle_connect 0x7fce440ae930 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900728318 0 0) 0x7fce440893c0 con 0x7fce440ae930 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:20.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:20.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:20.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:54:20.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.893+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce44088e00 con 0x7fce440ae930 2024-03-29T10:54:20.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941719204 0 0) 0x7fce44088e00 con 0x7fce440ae930 2024-03-29T10:54:20.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce440893c0 con 0x7fce440ae930 2024-03-29T10:54:20.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fce340017b0 con 0x7fce440ae930 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 10 monclient(hunting): dump: 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:20.878 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3888081283 0 0) 0x7fce440893c0 con 0x7fce440ae930 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 >> v1:172.21.15.174:6789/0 conn(0x7fce440acab0 legacy=0x7fce441efc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.894+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 >> v1:172.21.15.64:6789/0 conn(0x7fce440a9750 legacy=0x7fce4414dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 1 monclient: found mon.c 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce44087b60 con 0x7fce440ae930 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 10 monclient: _renew_subs 2024-03-29T10:54:20.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:20.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4414e700 con 0x7fce440ae930 2024-03-29T10:54:20.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:20.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 10 monclient: _renew_subs 2024-03-29T10:54:20.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:20.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce441f0530 con 0x7fce440ae930 2024-03-29T10:54:20.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.895+0000 7fce4ac99640 1 librados: init done 2024-03-29T10:54:20.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.898+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce34001910 con 0x7fce440ae930 2024-03-29T10:54:20.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:20.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3b7fe640 4 set_mon_vals no callback set 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce4ac99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fce34003790 con 0x7fce440ae930 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3a7fc640 10 monclient: dump: 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:20.882 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:20.883 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:20.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.899+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fce340197a0 con 0x7fce440ae930 2024-03-29T10:54:20.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce4ac99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:20.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce4ac99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:20.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c005080 con 0x7fce440ae930 2024-03-29T10:54:20.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fce34083810 con 0x7fce440ae930 2024-03-29T10:54:20.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce3a7fc640 10 client.4559.objecter ms_dispatch 0x7fce44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:20.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce3a7fc640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:20.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce3a7fc640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:20.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.900+0000 7fce3a7fc640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T10:54:20.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.904+0000 7fce49c97640 10 monclient: get_auth_request con 0x7fce2806de70 auth_method 0 2024-03-29T10:54:20.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.907+0000 7fce3a7fc640 10 client.4559.objecter ms_handle_connect 0x7fce2806de70 2024-03-29T10:54:20.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.908+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce3404f170 con 0x7fce440ae930 2024-03-29T10:54:20.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.908+0000 7fce3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:20.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:20.908+0000 7fce3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:21.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.241+0000 7fce4ac99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:21.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.241+0000 7fce4ac99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:21.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.241+0000 7fce4ac99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:21.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.241+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce2c005330 con 0x7fce440ae930 2024-03-29T10:54:21.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.242+0000 7fce3a7fc640 1 -- 172.21.15.64:0/1389819046 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fce340510a0 con 0x7fce440ae930 2024-03-29T10:54:21.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.242+0000 7fce3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:21.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.242+0000 7fce3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:21.224 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:21.224 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:21.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.246+0000 7fce4ac99640 10 librados: watch_flush enter 2024-03-29T10:54:21.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.246+0000 7fce4ac99640 10 librados: watch_flush exit 2024-03-29T10:54:21.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.246+0000 7fce4ac99640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T10:54:21.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.246+0000 7fce4ac99640 10 client.4559.objecter successfully canceled tick 2024-03-29T10:54:21.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.247+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 >> v1:172.21.15.174:6816/2473791532 conn(0x7fce2806de70 legacy=0x7fce28070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:21.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.247+0000 7fce4ac99640 10 monclient: shutdown 2024-03-29T10:54:21.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.247+0000 7fce4ac99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:21.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.247+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 >> v1:172.21.15.64:6790/0 conn(0x7fce440ae930 legacy=0x7fce441f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:21.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.247+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 shutdown_connections 2024-03-29T10:54:21.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.247+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 >> 172.21.15.64:0/1389819046 conn(0x7fce440673e0 msgr2=0x7fce44069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:21.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.249+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 shutdown_connections 2024-03-29T10:54:21.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.249+0000 7fce4ac99640 1 -- 172.21.15.64:0/1389819046 wait complete. 2024-03-29T10:54:21.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:21.249+0000 7fce4ac99640 1 librados: shutdown 2024-03-29T10:54:21.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:28.060 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.230+0000 7f2021066640 10 monclient: get_monmap_and_config 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.230+0000 7f2021066640 10 monclient: build_initial_monmap 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.230+0000 7f2021066640 10 monclient: monmap: 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:28.231428+0000 2024-03-29T10:54:28.213 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:28.231428+0000 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.231+0000 7f2021066640 1 Processor -- start 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.232+0000 7f2021066640 1 -- start start 2024-03-29T10:54:28.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.232+0000 7f2021066640 10 monclient: init 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient(hunting): picked mon.noname-b con 0x7f20140e2d10 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient(hunting): picked mon.noname-c con 0x7f201402b090 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient(hunting): picked mon.noname-a con 0x7f2014006940 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201402ac40 con 0x7f2014006940 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2014006dd0 con 0x7f201402b090 2024-03-29T10:54:28.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201402a430 con 0x7f20140e2d10 2024-03-29T10:54:28.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:28.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.233+0000 7f2021066640 10 monclient(hunting): authenticate will time out at 1279.532277s 2024-03-29T10:54:28.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201b7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f201402b090 0x7f2014029c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:48187/0 (socket says 172.21.15.64:48187) 2024-03-29T10:54:28.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201b7fe640 1 -- 172.21.15.64:0/2288590517 learned_addr learned my addr 172.21.15.64:0/2288590517 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:28.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598429472 0 0) 0x7f2014006dd0 con 0x7f201402b090 2024-03-29T10:54:28.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201affd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:28.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:28.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:28.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201affd640 10 monclient(hunting): my global_id is 4537 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.239+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f200c003520 con 0x7f201402b090 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94634277 0 0) 0x7f201402a430 con 0x7f20140e2d10 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2014006dd0 con 0x7f20140e2d10 2024-03-29T10:54:28.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389735828 0 0) 0x7f2014006dd0 con 0x7f20140e2d10 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 >> v1:172.21.15.174:6789/0 conn(0x7f201402b090 legacy=0x7f2014029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.240+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 >> v1:172.21.15.64:6789/0 conn(0x7f2014006940 legacy=0x7f20140f0ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 1 monclient: found mon.noname-b 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201402a5c0 con 0x7f20140e2d10 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 10 monclient: _finish_auth 0 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f201c04f730 con 0x7f20140e2d10 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.241+0000 7f201affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:28.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 10 monclient: dump: 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:28.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f201c0008d0 con 0x7f20140e2d10 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 1 -- 172.21.15.64:0/2288590517 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f201c0758b0 con 0x7f20140e2d10 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.242+0000 7f201affd640 10 monclient: dump: 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:28.228 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:28.229 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:28.229 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:28.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.244+0000 7f2021066640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:54:28.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.244+0000 7f2021066640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:28.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.244+0000 7f2021066640 4 set_mon_vals no callback set 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 10 monclient: shutdown 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 1 -- 172.21.15.64:0/2288590517 >> v1:172.21.15.64:6790/0 conn(0x7f20140e2d10 legacy=0x7f20140e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 1 -- 172.21.15.64:0/2288590517 shutdown_connections 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 1 -- 172.21.15.64:0/2288590517 >> 172.21.15.64:0/2288590517 conn(0x7f20140058f0 msgr2=0x7f2014005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 1 -- 172.21.15.64:0/2288590517 shutdown_connections 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.245+0000 7f2021066640 1 -- 172.21.15.64:0/2288590517 wait complete. 2024-03-29T10:54:28.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.246+0000 7f2021066640 10 monclient: build_initial_monmap 2024-03-29T10:54:28.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.246+0000 7f2021066640 10 monclient: monmap: 2024-03-29T10:54:28.232 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:28.232 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:28.232 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:28.232 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:28.232 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.246+0000 7f2021066640 1 librados: starting msgr at 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.246+0000 7f2021066640 1 librados: starting objecter 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.246+0000 7f2021066640 1 Processor -- start 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 1 -- start start 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 1 librados: setting wanted keys 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 1 librados: calling monclient init 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 10 monclient: init 2024-03-29T10:54:28.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.247+0000 7f2021066640 10 monclient(hunting): picked mon.noname-c con 0x7f2014006940 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 10 monclient(hunting): picked mon.noname-a con 0x7f201402b090 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 10 monclient(hunting): picked mon.noname-b con 0x7f20140e2d10 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2014034650 con 0x7f201402b090 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2014035010 con 0x7f20140e2d10 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2014035280 con 0x7f2014006940 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:28.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.248+0000 7f2021066640 10 monclient(hunting): authenticate will time out at 1279.546915s 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f201bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2014006940 0x7f2014033680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47507/0 (socket says 172.21.15.64:47507) 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f201bfff640 1 -- 172.21.15.64:0/889550968 learned_addr learned my addr 172.21.15.64:0/889550968 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 client.?.objecter ms_handle_connect 0x7f2014006940 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 client.?.objecter ms_handle_connect 0x7f201402b090 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626721230 0 0) 0x7f2014035280 con 0x7f2014006940 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:28.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.255+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2000004eb0 con 0x7f2014006940 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370149206 0 0) 0x7f2014034650 con 0x7f201402b090 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20000051d0 con 0x7f201402b090 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612059514 0 0) 0x7f2000004eb0 con 0x7f2014006940 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2000005640 con 0x7f2014006940 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f201c04f400 con 0x7f2014006940 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:28.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:28.240 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.256+0000 7f2018ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 136624623 0 0) 0x7f2000005640 con 0x7f2014006940 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 >> v1:172.21.15.174:6789/0 conn(0x7f20140e2d10 legacy=0x7f201418d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 >> v1:172.21.15.64:6789/0 conn(0x7f201402b090 legacy=0x7f2014033d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 1 monclient: found mon.c 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20140ddbb0 con 0x7f2014006940 2024-03-29T10:54:28.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 10 monclient: _finish_auth 0 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f201c04f730 con 0x7f2014006940 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f2018ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.258+0000 7f201a7fc640 4 set_mon_vals no callback set 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.259+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f201c04f400 con 0x7f2014006940 2024-03-29T10:54:28.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.259+0000 7f2018ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.259+0000 7f2018ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.259+0000 7f2018ff9640 10 monclient: dump: 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 5 monclient: authenticate success, global_id 4565 2024-03-29T10:54:28.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 10 monclient: _renew_subs 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20140d6a70 con 0x7f2014006940 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 10 monclient: _renew_subs 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20140d98e0 con 0x7f2014006940 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.260+0000 7f2021066640 1 librados: init done 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.262+0000 7f2021066640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.262+0000 7f2021066640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.262+0000 7f2021066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:28.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.262+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201402b620 con 0x7f2014006940 2024-03-29T10:54:28.248 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.267+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f201c067bf0 con 0x7f2014006940 2024-03-29T10:54:28.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.270+0000 7f201b7fe640 10 monclient: get_auth_request con 0x7f200006e260 auth_method 0 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.270+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f201c0bde10 con 0x7f2014006940 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.270+0000 7f2018ff9640 10 client.4565.objecter ms_dispatch 0x7f201c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.270+0000 7f2018ff9640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.270+0000 7f2018ff9640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.270+0000 7f2018ff9640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.271+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f201c0ba0b0 con 0x7f2014006940 2024-03-29T10:54:28.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.271+0000 7f2018ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:28.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.271+0000 7f2018ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:28.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.271+0000 7f2018ff9640 10 client.4565.objecter ms_handle_connect 0x7f200006e260 2024-03-29T10:54:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.623+0000 7f2021066640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.623+0000 7f2021066640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:28.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.623+0000 7f2021066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.623+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2014004cd0 con 0x7f2014006940 2024-03-29T10:54:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.624+0000 7f2018ff9640 1 -- 172.21.15.64:0/889550968 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f201c0ba0b0 con 0x7f2014006940 2024-03-29T10:54:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.624+0000 7f2018ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:28.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.624+0000 7f2018ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:28.607 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:28.607 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.631+0000 7f2021066640 10 librados: watch_flush enter 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.631+0000 7f2021066640 10 librados: watch_flush exit 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.631+0000 7f2021066640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.631+0000 7f2021066640 10 client.4565.objecter successfully canceled tick 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.632+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 >> v1:172.21.15.174:6816/2473791532 conn(0x7f200006e260 legacy=0x7f2000070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.632+0000 7f2021066640 10 monclient: shutdown 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.632+0000 7f2021066640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.632+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 >> v1:172.21.15.64:6790/0 conn(0x7f2014006940 legacy=0x7f2014033680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:28.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.632+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 shutdown_connections 2024-03-29T10:54:28.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.632+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 >> 172.21.15.64:0/889550968 conn(0x7f20140058f0 msgr2=0x7f20140e4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:28.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.633+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 shutdown_connections 2024-03-29T10:54:28.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.633+0000 7f2021066640 1 -- 172.21.15.64:0/889550968 wait complete. 2024-03-29T10:54:28.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:28.633+0000 7f2021066640 1 librados: shutdown 2024-03-29T10:54:28.632 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:35.438 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.603+0000 7f0a2acb3640 10 monclient: get_monmap_and_config 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.603+0000 7f0a2acb3640 10 monclient: build_initial_monmap 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.603+0000 7f0a2acb3640 10 monclient: monmap: 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:35.604165+0000 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:35.604165+0000 2024-03-29T10:54:35.584 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.604+0000 7f0a2acb3640 1 Processor -- start 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.604+0000 7f0a2acb3640 1 -- start start 2024-03-29T10:54:35.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.604+0000 7f0a2acb3640 10 monclient: init 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient(hunting): picked mon.noname-a con 0x7f0a240afae0 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient(hunting): picked mon.noname-c con 0x7f0a240b0400 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2406fe70 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a240726d0 con 0x7f0a240afae0 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2413be10 con 0x7f0a240b0400 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2413bfa0 con 0x7f0a2406fe70 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:35.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.605+0000 7f0a2acb3640 10 monclient(hunting): authenticate will time out at 1286.904655s 2024-03-29T10:54:35.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.607+0000 7f0a29cb1640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0a240afae0 0x7f0a24071480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33920/0 (socket says 172.21.15.64:33920) 2024-03-29T10:54:35.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.607+0000 7f0a29cb1640 1 -- 172.21.15.64:0/1247936073 learned_addr learned my addr 172.21.15.64:0/1247936073 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:35.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.608+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200201567 0 0) 0x7f0a240726d0 con 0x7f0a240afae0 2024-03-29T10:54:35.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.608+0000 7f0a28caf640 10 monclient(hunting): _negotiate 2024-03-29T10:54:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.608+0000 7f0a28caf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.608+0000 7f0a28caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.608+0000 7f0a28caf640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:54:35.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.608+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a10003520 con 0x7f0a240afae0 2024-03-29T10:54:35.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.609+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910316207 0 0) 0x7f0a10003520 con 0x7f0a240afae0 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 >> v1:172.21.15.64:6790/0 conn(0x7f0a2406fe70 legacy=0x7f0a2413b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 >> v1:172.21.15.174:6789/0 conn(0x7f0a240b0400 legacy=0x7f0a24072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: _finish_hunting 0 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 1 monclient: found mon.noname-a 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2406f360 con 0x7f0a240afae0 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: _finish_auth 0 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: _check_auth_tickets 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0a20001690 con 0x7f0a240afae0 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:35.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a28caf640 10 monclient: dump: 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a2acb3640 5 monclient: authenticate success, global_id 4512 2024-03-29T10:54:35.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.611+0000 7f0a2acb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a20001860 con 0x7f0a240afae0 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a28caf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a28caf640 1 -- 172.21.15.64:0/1247936073 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0a20001c70 con 0x7f0a240afae0 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a28caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a28caf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a28caf640 10 monclient: dump: 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.595 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a2acb3640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:35.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.612+0000 7f0a2acb3640 4 set_mon_vals no callback set 2024-03-29T10:54:35.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.613+0000 7f0a2acb3640 10 monclient: shutdown 2024-03-29T10:54:35.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.613+0000 7f0a2acb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:35.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.613+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1247936073 >> v1:172.21.15.64:6789/0 conn(0x7f0a240afae0 legacy=0x7f0a24071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:35.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.616+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1247936073 shutdown_connections 2024-03-29T10:54:35.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.616+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1247936073 >> 172.21.15.64:0/1247936073 conn(0x7f0a240b0960 msgr2=0x7f0a240b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:35.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.616+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1247936073 shutdown_connections 2024-03-29T10:54:35.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.616+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1247936073 wait complete. 2024-03-29T10:54:35.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.617+0000 7f0a2acb3640 10 monclient: build_initial_monmap 2024-03-29T10:54:35.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.618+0000 7f0a2acb3640 10 monclient: monmap: 2024-03-29T10:54:35.598 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:35.599 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:35.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.618+0000 7f0a2acb3640 1 librados: starting msgr at 2024-03-29T10:54:35.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.618+0000 7f0a2acb3640 1 librados: starting objecter 2024-03-29T10:54:35.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.618+0000 7f0a2acb3640 1 Processor -- start 2024-03-29T10:54:35.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.619+0000 7f0a2acb3640 1 -- start start 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.619+0000 7f0a2acb3640 1 librados: setting wanted keys 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.619+0000 7f0a2acb3640 1 librados: calling monclient init 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.619+0000 7f0a2acb3640 10 monclient: init 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2406fe70 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient(hunting): picked mon.noname-c con 0x7f0a240afae0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2413db40 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a241d6da0 con 0x7f0a2413db40 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2413f2b0 con 0x7f0a2406fe70 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a24134c60 con 0x7f0a240afae0 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:35.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.620+0000 7f0a2acb3640 10 monclient(hunting): authenticate will time out at 1286.919688s 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a2a4b2640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0a2413db40 0x7f0a241dd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38032/0 (socket says 172.21.15.64:38032) 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a2a4b2640 1 -- 172.21.15.64:0/1173457875 learned_addr learned my addr 172.21.15.64:0/1173457875 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 10 client.?.objecter ms_handle_connect 0x7f0a2413db40 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346687005 0 0) 0x7f0a241d6da0 con 0x7f0a2413db40 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.621+0000 7f0a1a7fc640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:54:35.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.622+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a10003520 con 0x7f0a2413db40 2024-03-29T10:54:35.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 157257334 0 0) 0x7f0a10003520 con 0x7f0a2413db40 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a241d6da0 con 0x7f0a2413db40 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0a1c001c60 con 0x7f0a2413db40 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient(hunting): dump: 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:35.610 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 880357052 0 0) 0x7f0a241d6da0 con 0x7f0a2413db40 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 >> v1:172.21.15.64:6790/0 conn(0x7f0a240afae0 legacy=0x7f0a2413d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 >> v1:172.21.15.174:6789/0 conn(0x7f0a2406fe70 legacy=0x7f0a241d6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 monclient: found mon.a 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2413dfd0 con 0x7f0a2413db40 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:35.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 5 monclient: authenticate success, global_id 4515 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 10 monclient: _renew_subs 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a241d68a0 con 0x7f0a2413db40 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 10 monclient: _renew_subs 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a241ded60 con 0x7f0a2413db40 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a2acb3640 1 librados: init done 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a1c003970 con 0x7f0a2413db40 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.626+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0a1c003d80 con 0x7f0a2413db40 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.627+0000 7f0a1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:35.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.627+0000 7f0a1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.627+0000 7f0a1a7fc640 10 monclient: dump: 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.628+0000 7f0a2acb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:35.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.628+0000 7f0a1b7fe640 4 set_mon_vals no callback set 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f0a1c003ee0 con 0x7f0a2413db40 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f0a1c084e60 con 0x7f0a2413db40 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a1a7fc640 10 client.4515.objecter ms_dispatch 0x7f0a24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a1a7fc640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a1a7fc640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a1a7fc640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T10:54:35.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.630+0000 7f0a2acb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:35.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.631+0000 7f0a2acb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:35.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.631+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2413f9e0 con 0x7f0a2413db40 2024-03-29T10:54:35.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.631+0000 7f0a29cb1640 10 monclient: get_auth_request con 0x7f0a10072250 auth_method 0 2024-03-29T10:54:35.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.632+0000 7f0a1a7fc640 10 client.4515.objecter ms_handle_connect 0x7f0a10072250 2024-03-29T10:54:35.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.638+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a1c050740 con 0x7f0a2413db40 2024-03-29T10:54:35.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.638+0000 7f0a1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:35.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.638+0000 7f0a1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.968+0000 7f0a2acb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.968+0000 7f0a2acb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.968+0000 7f0a2acb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:54:35.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.968+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a2404f730 con 0x7f0a2413db40 2024-03-29T10:54:35.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.969+0000 7f0a1a7fc640 1 -- 172.21.15.64:0/1173457875 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f0a1c052670 con 0x7f0a2413db40 2024-03-29T10:54:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.969+0000 7f0a1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:35.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.969+0000 7f0a1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:35.953 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:35.953 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 10 librados: watch_flush enter 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 10 librados: watch_flush exit 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 10 client.4515.objecter successfully canceled tick 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 >> v1:172.21.15.174:6816/2473791532 conn(0x7f0a10072250 legacy=0x7f0a10074710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 10 monclient: shutdown 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:35.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.975+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 >> v1:172.21.15.64:6789/0 conn(0x7f0a2413db40 legacy=0x7f0a241dd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:35.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.977+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 shutdown_connections 2024-03-29T10:54:35.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.977+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 >> 172.21.15.64:0/1173457875 conn(0x7f0a240b0960 msgr2=0x7f0a24069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.977+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 shutdown_connections 2024-03-29T10:54:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.977+0000 7f0a2acb3640 1 -- 172.21.15.64:0/1173457875 wait complete. 2024-03-29T10:54:35.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:35.977+0000 7f0a2acb3640 1 librados: shutdown 2024-03-29T10:54:35.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:42.782 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.937+0000 7f4e73a57640 10 monclient: get_monmap_and_config 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.937+0000 7f4e73a57640 10 monclient: build_initial_monmap 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.937+0000 7f4e73a57640 10 monclient: monmap: 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:42.938237+0000 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:42.938237+0000 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:42.919 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:42.920 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:42.920 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.938+0000 7f4e73a57640 1 Processor -- start 2024-03-29T10:54:42.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.938+0000 7f4e73a57640 1 -- start start 2024-03-29T10:54:42.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.938+0000 7f4e73a57640 10 monclient: init 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient(hunting): picked mon.noname-c con 0x7f4e6c071de0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient(hunting): picked mon.noname-a con 0x7f4e6c0721c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6c145ae0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c145650 con 0x7f4e6c0721c0 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c0ae260 con 0x7f4e6c071de0 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c1420d0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:42.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e73a57640 10 monclient(hunting): authenticate will time out at 1294.238265s 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e71ff4640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4e6c145ae0 0x7f4e6c145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:43944/0 (socket says 172.21.15.64:43944) 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e71ff4640 1 -- 172.21.15.64:0/3112607731 learned_addr learned my addr 172.21.15.64:0/3112607731 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013965320 0 0) 0x7f4e6c1420d0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e63fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.939+0000 7f4e63fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.940+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e58003520 con 0x7f4e6c145ae0 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.940+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236425974 0 0) 0x7f4e58003520 con 0x7f4e6c145ae0 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.940+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 >> v1:172.21.15.174:6789/0 conn(0x7f4e6c071de0 legacy=0x7f4e6c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:42.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.940+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 >> v1:172.21.15.64:6789/0 conn(0x7f4e6c0721c0 legacy=0x7f4e6c13f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 1 monclient: found mon.noname-b 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0b0590 con 0x7f4e6c145ae0 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e64001a10 con 0x7f4e6c145ae0 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.942+0000 7f4e63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:42.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 10 monclient: dump: 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e73a57640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e73a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:42.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e64001b70 con 0x7f4e6c145ae0 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 1 -- 172.21.15.64:0/3112607731 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e64001f40 con 0x7f4e6c145ae0 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e63fff640 10 monclient: dump: 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:42.926 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:42.927 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:42.927 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e73a57640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:42.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.943+0000 7f4e73a57640 4 set_mon_vals no callback set 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 10 monclient: shutdown 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 1 -- 172.21.15.64:0/3112607731 >> v1:172.21.15.64:6790/0 conn(0x7f4e6c145ae0 legacy=0x7f4e6c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 1 -- 172.21.15.64:0/3112607731 shutdown_connections 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 1 -- 172.21.15.64:0/3112607731 >> 172.21.15.64:0/3112607731 conn(0x7f4e6c06b660 msgr2=0x7f4e6c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 1 -- 172.21.15.64:0/3112607731 shutdown_connections 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.944+0000 7f4e73a57640 1 -- 172.21.15.64:0/3112607731 wait complete. 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.945+0000 7f4e73a57640 10 monclient: build_initial_monmap 2024-03-29T10:54:42.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.945+0000 7f4e73a57640 10 monclient: monmap: 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 1 librados: starting msgr at 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 1 librados: starting objecter 2024-03-29T10:54:42.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 1 Processor -- start 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 1 -- start start 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 1 librados: setting wanted keys 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 1 librados: calling monclient init 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.946+0000 7f4e73a57640 10 monclient: init 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6c071de0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 10 monclient(hunting): picked mon.noname-c con 0x7f4e6c145ae0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e70ff2640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4e6c145ae0 0x7f4e6c134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45688/0 (socket says 172.21.15.64:45688) 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e70ff2640 1 -- 172.21.15.64:0/343101452 learned_addr learned my addr 172.21.15.64:0/343101452 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 10 monclient(hunting): picked mon.noname-a con 0x7f4e6c136830 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c139c90 con 0x7f4e6c136830 2024-03-29T10:54:42.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e61ffb640 10 client.?.objecter ms_handle_connect 0x7f4e6c145ae0 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c135170 con 0x7f4e6c071de0 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c13b400 con 0x7f4e6c145ae0 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e61ffb640 10 client.?.objecter ms_handle_connect 0x7f4e6c071de0 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.947+0000 7f4e73a57640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e73a57640 10 monclient(hunting): authenticate will time out at 1294.246911s 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459483392 0 0) 0x7f4e6c13b400 con 0x7f4e6c145ae0 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:42.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e50004eb0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327214281 0 0) 0x7f4e6c135170 con 0x7f4e6c071de0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.948+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e6c13b400 con 0x7f4e6c071de0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3522547479 0 0) 0x7f4e6c13b400 con 0x7f4e6c071de0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e6c135170 con 0x7f4e6c071de0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1393729067 0 0) 0x7f4e50004eb0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e6c13b400 con 0x7f4e6c145ae0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e68001890 con 0x7f4e6c071de0 2024-03-29T10:54:42.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.949+0000 7f4e61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e5c001910 con 0x7f4e6c145ae0 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:42.939 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1648149264 0 0) 0x7f4e6c13b400 con 0x7f4e6c145ae0 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 >> v1:172.21.15.174:6789/0 conn(0x7f4e6c071de0 legacy=0x7f4e6c13a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.950+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 >> v1:172.21.15.64:6789/0 conn(0x7f4e6c136830 legacy=0x7f4e6c134a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 1 monclient: found mon.c 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c13b6e0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:42.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e73a57640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e73a57640 10 monclient: _renew_subs 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e73a57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.951+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c13b9c0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e73a57640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e73a57640 10 monclient: _renew_subs 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e73a57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c13dea0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e73a57640 1 librados: init done 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e5c0035f0 con 0x7f4e6c145ae0 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4e5c001a70 con 0x7f4e6c145ae0 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e61ffb640 10 monclient: dump: 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:42.943 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.952+0000 7f4e637fe640 4 set_mon_vals no callback set 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.954+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f4e5c003a40 con 0x7f4e6c145ae0 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.954+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f4e5c084960 con 0x7f4e6c145ae0 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.954+0000 7f4e61ffb640 10 client.4571.objecter ms_dispatch 0x7f4e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.954+0000 7f4e61ffb640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:42.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.954+0000 7f4e61ffb640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e61ffb640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e73a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e73a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e73a57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c005080 con 0x7f4e6c145ae0 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e717f3640 10 monclient: get_auth_request con 0x7f4e5006def0 auth_method 0 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.955+0000 7f4e61ffb640 10 client.4571.objecter ms_handle_connect 0x7f4e5006def0 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.964+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e5c050240 con 0x7f4e6c145ae0 2024-03-29T10:54:42.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.964+0000 7f4e61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:42.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:42.964+0000 7f4e61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:43.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.292+0000 7f4e73a57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:43.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.292+0000 7f4e73a57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:43.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.292+0000 7f4e73a57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:43.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.292+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e4c005330 con 0x7f4e6c145ae0 2024-03-29T10:54:43.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.294+0000 7f4e61ffb640 1 -- 172.21.15.64:0/343101452 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f4e5c052170 con 0x7f4e6c145ae0 2024-03-29T10:54:43.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.294+0000 7f4e61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:43.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.294+0000 7f4e61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:43.276 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:43.276 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:43.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.298+0000 7f4e73a57640 10 librados: watch_flush enter 2024-03-29T10:54:43.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 10 librados: watch_flush exit 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 10 client.4571.objecter successfully canceled tick 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 >> v1:172.21.15.174:6816/2473791532 conn(0x7f4e5006def0 legacy=0x7f4e500703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 10 monclient: shutdown 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 >> v1:172.21.15.64:6790/0 conn(0x7f4e6c145ae0 legacy=0x7f4e6c134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 shutdown_connections 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.301+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 >> 172.21.15.64:0/343101452 conn(0x7f4e6c06b660 msgr2=0x7f4e6c0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.302+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 shutdown_connections 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.302+0000 7f4e73a57640 1 -- 172.21.15.64:0/343101452 wait complete. 2024-03-29T10:54:43.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:43.302+0000 7f4e73a57640 1 librados: shutdown 2024-03-29T10:54:43.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:50.109 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:50.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.278+0000 7f89521ef640 10 monclient: get_monmap_and_config 2024-03-29T10:54:50.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.278+0000 7f89521ef640 10 monclient: build_initial_monmap 2024-03-29T10:54:50.261 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.278+0000 7f89521ef640 10 monclient: monmap: 2024-03-29T10:54:50.261 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:50.261 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:50.261 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:50.280440+0000 2024-03-29T10:54:50.262 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:50.280440+0000 2024-03-29T10:54:50.262 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:50.262 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:50.262 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:50.263 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:50.263 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:50.263 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:50.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.279+0000 7f89521ef640 1 Processor -- start 2024-03-29T10:54:50.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.279+0000 7f89521ef640 1 -- start start 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.279+0000 7f89521ef640 10 monclient: init 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient(hunting): picked mon.noname-a con 0x7f894c0b4330 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient(hunting): picked mon.noname-c con 0x7f894c1470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient(hunting): picked mon.noname-b con 0x7f894c1439b0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c06f2a0 con 0x7f894c0b4330 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c146b90 con 0x7f894c1470b0 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c0b2ca0 con 0x7f894c1439b0 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:50.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f89521ef640 10 monclient(hunting): authenticate will time out at 1301.580631s 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f894b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f894c0b4330 0x7f894c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41754/0 (socket says 172.21.15.64:41754) 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.280+0000 7f894b7fe640 1 -- 172.21.15.64:0/314314659 learned_addr learned my addr 172.21.15.64:0/314314659 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594200010 0 0) 0x7f894c0b2ca0 con 0x7f894c1439b0 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934003520 con 0x7f894c1439b0 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113260374 0 0) 0x7f894c06f2a0 con 0x7f894c0b4330 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:50.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.281+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c0b2ca0 con 0x7f894c0b4330 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256240889 0 0) 0x7f894c146b90 con 0x7f894c1470b0 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient(hunting): my global_id is 4546 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c06f2a0 con 0x7f894c1470b0 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633936762 0 0) 0x7f8934003520 con 0x7f894c1439b0 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 >> v1:172.21.15.174:6789/0 conn(0x7f894c1470b0 legacy=0x7f894c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 >> v1:172.21.15.64:6789/0 conn(0x7f894c0b4330 legacy=0x7f894c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:50.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:54:50.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:50.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:54:50.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c06e0c0 con 0x7f894c1439b0 2024-03-29T10:54:50.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:50.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:50.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8940001690 con 0x7f894c1439b0 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f894a7fc640 10 monclient: dump: 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:50.278 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.279 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.279 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:50.279 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:50.279 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:50.279 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:50.279 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f89521ef640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.282+0000 7f89521ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8940001870 con 0x7f894c1439b0 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f894a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f89521ef640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f89521ef640 4 set_mon_vals no callback set 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f894a7fc640 1 -- 172.21.15.64:0/314314659 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8940001690 con 0x7f894c1439b0 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f894a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f894a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:50.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f894a7fc640 10 monclient: dump: 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f89521ef640 10 monclient: shutdown 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f89521ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:50.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.283+0000 7f89521ef640 1 -- 172.21.15.64:0/314314659 >> v1:172.21.15.64:6790/0 conn(0x7f894c1439b0 legacy=0x7f894c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.284+0000 7f89521ef640 1 -- 172.21.15.64:0/314314659 shutdown_connections 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.284+0000 7f89521ef640 1 -- 172.21.15.64:0/314314659 >> 172.21.15.64:0/314314659 conn(0x7f894c066a00 msgr2=0x7f894c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.284+0000 7f89521ef640 1 -- 172.21.15.64:0/314314659 shutdown_connections 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.284+0000 7f89521ef640 1 -- 172.21.15.64:0/314314659 wait complete. 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.284+0000 7f89521ef640 10 monclient: build_initial_monmap 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.284+0000 7f89521ef640 10 monclient: monmap: 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:50.282 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 1 librados: starting msgr at 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 1 librados: starting objecter 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 1 Processor -- start 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 1 -- start start 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 1 librados: setting wanted keys 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 1 librados: calling monclient init 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.285+0000 7f89521ef640 10 monclient: init 2024-03-29T10:54:50.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 10 monclient(hunting): picked mon.noname-c con 0x7f894c0b4330 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 10 monclient(hunting): picked mon.noname-b con 0x7f894c1439b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 10 monclient(hunting): picked mon.noname-a con 0x7f894c1470b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c1dfdc0 con 0x7f894c1470b0 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f894b7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f894c0b4330 0x7f894c1df140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33518/0 (socket says 172.21.15.64:33518) 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c1df850 con 0x7f894c1439b0 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f894b7fe640 1 -- 172.21.15.64:0/3592968554 learned_addr learned my addr 172.21.15.64:0/3592968554 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.286+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c1df9e0 con 0x7f894c0b4330 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f89521ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f89521ef640 10 monclient(hunting): authenticate will time out at 1301.587014s 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 client.?.objecter ms_handle_connect 0x7f894c0b4330 2024-03-29T10:54:50.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 client.?.objecter ms_handle_connect 0x7f894c1439b0 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 client.?.objecter ms_handle_connect 0x7f894c1470b0 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050357405 0 0) 0x7f894c1df9e0 con 0x7f894c0b4330 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8938004eb0 con 0x7f894c0b4330 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226392939 0 0) 0x7f894c1dfdc0 con 0x7f894c1470b0 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:50.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c1df9e0 con 0x7f894c1470b0 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646631460 0 0) 0x7f894c1df850 con 0x7f894c1439b0 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 10 monclient(hunting): my global_id is 4549 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.287+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c1dfdc0 con 0x7f894c1439b0 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983951868 0 0) 0x7f8938004eb0 con 0x7f894c0b4330 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f894c1df850 con 0x7f894c0b4330 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8944001c30 con 0x7f894c0b4330 2024-03-29T10:54:50.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient(hunting): dump: 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:50.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 464310991 0 0) 0x7f894c1df850 con 0x7f894c0b4330 2024-03-29T10:54:50.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 >> v1:172.21.15.174:6789/0 conn(0x7f894c1439b0 legacy=0x7f894c1253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 >> v1:172.21.15.64:6789/0 conn(0x7f894c1470b0 legacy=0x7f894c125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:50.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 monclient: found mon.c 2024-03-29T10:54:50.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:50.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c140ee0 con 0x7f894c0b4330 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f8933fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f89521ef640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f89521ef640 10 monclient: _renew_subs 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f89521ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.288+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c141190 con 0x7f894c0b4330 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f89521ef640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:50.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f89521ef640 10 monclient: _renew_subs 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f89521ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c1e7690 con 0x7f894c0b4330 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f89521ef640 1 librados: init done 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8944001d90 con 0x7f894c0b4330 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8933fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f8944004d40 con 0x7f894c0b4330 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8933fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8933fff640 10 monclient: dump: 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:50.290 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.289+0000 7f8949ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.290+0000 7f89521ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.290+0000 7f89521ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.290+0000 7f89521ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:50.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.290+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c005080 con 0x7f894c0b4330 2024-03-29T10:54:50.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.292+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f8944004ea0 con 0x7f894c0b4330 2024-03-29T10:54:50.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.293+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f8944084d30 con 0x7f894c0b4330 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.293+0000 7f8933fff640 10 client.4577.objecter ms_dispatch 0x7f894c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.293+0000 7f8933fff640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.293+0000 7f8933fff640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.293+0000 7f8933fff640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.293+0000 7f894affd640 10 monclient: get_auth_request con 0x7f893806dea0 auth_method 0 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.294+0000 7f8933fff640 10 client.4577.objecter ms_handle_connect 0x7f893806dea0 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.300+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8944050440 con 0x7f894c0b4330 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.300+0000 7f8933fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:50.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.300+0000 7f8933fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.669+0000 7f89521ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.669+0000 7f89521ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.669+0000 7f89521ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:50.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.669+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f892c005330 con 0x7f894c0b4330 2024-03-29T10:54:50.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.670+0000 7f8933fff640 1 -- 172.21.15.64:0/3592968554 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f8944052540 con 0x7f894c0b4330 2024-03-29T10:54:50.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.670+0000 7f8933fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:50.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.670+0000 7f8933fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:50.653 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:50.653 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.673+0000 7f89521ef640 10 librados: watch_flush enter 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.673+0000 7f89521ef640 10 librados: watch_flush exit 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.673+0000 7f89521ef640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 10 client.4577.objecter successfully canceled tick 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 >> v1:172.21.15.174:6816/2473791532 conn(0x7f893806dea0 legacy=0x7f8938070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 10 monclient: shutdown 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 >> v1:172.21.15.64:6790/0 conn(0x7f894c0b4330 legacy=0x7f894c1df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:50.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 shutdown_connections 2024-03-29T10:54:50.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 >> 172.21.15.64:0/3592968554 conn(0x7f894c066a00 msgr2=0x7f894c0b82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:50.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 shutdown_connections 2024-03-29T10:54:50.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 -- 172.21.15.64:0/3592968554 wait complete. 2024-03-29T10:54:50.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:50.674+0000 7f89521ef640 1 librados: shutdown 2024-03-29T10:54:50.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:57.479 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:57.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.632+0000 7fac25401640 10 monclient: get_monmap_and_config 2024-03-29T10:54:57.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.632+0000 7fac25401640 10 monclient: build_initial_monmap 2024-03-29T10:54:57.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.632+0000 7fac25401640 10 monclient: monmap: 2024-03-29T10:54:57.614 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:57.614 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:54:57.634148+0000 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:54:57.634148+0000 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:54:57.615 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:57.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.633+0000 7fac25401640 1 Processor -- start 2024-03-29T10:54:57.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.633+0000 7fac25401640 1 -- start start 2024-03-29T10:54:57.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.633+0000 7fac25401640 10 monclient: init 2024-03-29T10:54:57.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.634+0000 7fac25401640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:57.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.634+0000 7fac25401640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:57.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.634+0000 7fac25401640 10 monclient(hunting): picked mon.noname-c con 0x7fac200b2800 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:57.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.634+0000 7fac25401640 10 monclient(hunting): picked mon.noname-b con 0x7fac200a9c00 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:57.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac25401640 10 monclient(hunting): picked mon.noname-a con 0x7fac200b38c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:57.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac25401640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac20141eb0 con 0x7fac200b38c0 2024-03-29T10:54:57.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac25401640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200b22d0 con 0x7fac200b2800 2024-03-29T10:54:57.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac25401640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac20142d30 con 0x7fac200a9c00 2024-03-29T10:54:57.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac25401640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:57.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac25401640 10 monclient(hunting): authenticate will time out at 1308.935157s 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac1f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fac200a9c00 0x7fac2013bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42206/0 (socket says 172.21.15.64:42206) 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac1f7fe640 1 -- 172.21.15.64:0/1987178282 learned_addr learned my addr 172.21.15.64:0/1987178282 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.635+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067401365 0 0) 0x7fac20142d30 con 0x7fac200a9c00 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac08003520 con 0x7fac200a9c00 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 222222060 0 0) 0x7fac08003520 con 0x7fac200a9c00 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 >> v1:172.21.15.174:6789/0 conn(0x7fac200b2800 legacy=0x7fac20142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:57.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.636+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 >> v1:172.21.15.64:6789/0 conn(0x7fac200b38c0 legacy=0x7fac2014d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 1 monclient: found mon.noname-b 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac200aaf90 con 0x7fac200a9c00 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: _finish_auth 0 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:57.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fac14001690 con 0x7fac200a9c00 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: dump: 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:57.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac25401640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac25401640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac140018b0 con 0x7fac200a9c00 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 1 -- 172.21.15.64:0/1987178282 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fac14001c40 con 0x7fac200a9c00 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac25401640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac25401640 4 set_mon_vals no callback set 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.638+0000 7fac1effd640 10 monclient: dump: 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.625 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 10 monclient: shutdown 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 1 -- 172.21.15.64:0/1987178282 >> v1:172.21.15.64:6790/0 conn(0x7fac200a9c00 legacy=0x7fac2013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 1 -- 172.21.15.64:0/1987178282 shutdown_connections 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 1 -- 172.21.15.64:0/1987178282 >> 172.21.15.64:0/1987178282 conn(0x7fac20066a00 msgr2=0x7fac20066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 1 -- 172.21.15.64:0/1987178282 shutdown_connections 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.639+0000 7fac25401640 1 -- 172.21.15.64:0/1987178282 wait complete. 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.640+0000 7fac25401640 10 monclient: build_initial_monmap 2024-03-29T10:54:57.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.640+0000 7fac25401640 10 monclient: monmap: 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 1 librados: starting msgr at 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 1 librados: starting objecter 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 1 Processor -- start 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 1 -- start start 2024-03-29T10:54:57.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 1 librados: setting wanted keys 2024-03-29T10:54:57.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 1 librados: calling monclient init 2024-03-29T10:54:57.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.641+0000 7fac25401640 10 monclient: init 2024-03-29T10:54:57.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 10 monclient(hunting): picked mon.noname-a con 0x7fac200a9c00 addr v1:172.21.15.64:6789/0 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 10 monclient(hunting): picked mon.noname-b con 0x7fac200b2800 addr v1:172.21.15.174:6789/0 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 10 monclient(hunting): picked mon.noname-c con 0x7fac200b38c0 addr v1:172.21.15.64:6790/0 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac201da5f0 con 0x7fac200a9c00 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200afca0 con 0x7fac200b2800 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac201db190 con 0x7fac200b38c0 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.642+0000 7fac25401640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.643+0000 7fac25401640 10 monclient(hunting): authenticate will time out at 1308.942892s 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.643+0000 7fac24c00640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fac200b38c0 0x7fac200b0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42738/0 (socket says 172.21.15.64:42738) 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.643+0000 7fac24c00640 1 -- 172.21.15.64:0/1065045952 learned_addr learned my addr 172.21.15.64:0/1065045952 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:54:57.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 10 client.?.objecter ms_handle_connect 0x7fac200b38c0 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445361716 0 0) 0x7fac201db190 con 0x7fac200b38c0 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.644+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac0c004eb0 con 0x7fac200b38c0 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 10 client.?.objecter ms_handle_connect 0x7fac200b2800 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137175478 0 0) 0x7fac0c004eb0 con 0x7fac200b38c0 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac201db190 con 0x7fac200b38c0 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fac18001ad0 con 0x7fac200b38c0 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:57.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 10 monclient(hunting): dump: 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2783518072 0 0) 0x7fac201db190 con 0x7fac200b38c0 2024-03-29T10:54:57.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 >> v1:172.21.15.174:6789/0 conn(0x7fac200b2800 legacy=0x7fac201daa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.645+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 >> v1:172.21.15.64:6789/0 conn(0x7fac200a9c00 legacy=0x7fac200ae740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 1 monclient: found mon.c 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac201db320 con 0x7fac200b38c0 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: _finish_auth 0 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 10 monclient: _renew_subs 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac200b0410 con 0x7fac200b38c0 2024-03-29T10:54:57.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 10 monclient: _renew_subs 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac200b0760 con 0x7fac200b38c0 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac25401640 1 librados: init done 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac18002060 con 0x7fac200b38c0 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1e7fc640 4 set_mon_vals no callback set 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fac18004be0 con 0x7fac200b38c0 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.646+0000 7fac1cff9640 10 monclient: dump: 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.636 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.648+0000 7fac25401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.648+0000 7fac25401640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.648+0000 7fac25401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.648+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac00005080 con 0x7fac200b38c0 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fac1801abb0 con 0x7fac200b38c0 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fac18084bc0 con 0x7fac200b38c0 2024-03-29T10:54:57.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1cff9640 10 client.4583.objecter ms_dispatch 0x7fac20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:57.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1cff9640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:57.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1cff9640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:57.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1cff9640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T10:54:57.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.649+0000 7fac1ffff640 10 monclient: get_auth_request con 0x7fac0c06db80 auth_method 0 2024-03-29T10:54:57.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.650+0000 7fac1cff9640 10 client.4583.objecter ms_handle_connect 0x7fac0c06db80 2024-03-29T10:54:57.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.657+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac18050500 con 0x7fac200b38c0 2024-03-29T10:54:57.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.657+0000 7fac1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:57.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.657+0000 7fac1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:57.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.970+0000 7fac25401640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:57.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.970+0000 7fac25401640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:57.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.970+0000 7fac25401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:54:57.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.970+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac00005330 con 0x7fac200b38c0 2024-03-29T10:54:57.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.971+0000 7fac1cff9640 1 -- 172.21.15.64:0/1065045952 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fac18052430 con 0x7fac200b38c0 2024-03-29T10:54:57.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.971+0000 7fac1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:57.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.971+0000 7fac1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:57.953 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:54:57.953 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:57.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 10 librados: watch_flush enter 2024-03-29T10:54:57.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 10 librados: watch_flush exit 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 10 client.4583.objecter successfully canceled tick 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 >> v1:172.21.15.174:6816/2473791532 conn(0x7fac0c06db80 legacy=0x7fac0c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 10 monclient: shutdown 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 >> v1:172.21.15.64:6790/0 conn(0x7fac200b38c0 legacy=0x7fac200b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 shutdown_connections 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 >> 172.21.15.64:0/1065045952 conn(0x7fac20066a00 msgr2=0x7fac2013bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 shutdown_connections 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.975+0000 7fac25401640 1 -- 172.21.15.64:0/1065045952 wait complete. 2024-03-29T10:54:57.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:54:57.976+0000 7fac25401640 1 librados: shutdown 2024-03-29T10:54:57.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:04.779 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:04.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.943+0000 7fcaaaa9d640 10 monclient: get_monmap_and_config 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.943+0000 7fcaaaa9d640 10 monclient: build_initial_monmap 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.943+0000 7fcaaaa9d640 10 monclient: monmap: 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:04.944419+0000 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:04.944419+0000 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:04.925 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:04.926 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:04.926 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.944+0000 7fcaaaa9d640 1 Processor -- start 2024-03-29T10:55:04.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.944+0000 7fcaaaa9d640 1 -- start start 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.944+0000 7fcaaaa9d640 10 monclient: init 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa41422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa4142690 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa40b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa4142c80 con 0x7fcaa4142690 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40aae40 con 0x7fcaa41422b0 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa413fbd0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:04.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.945+0000 7fcaaaa9d640 10 monclient(hunting): authenticate will time out at 1316.244465s 2024-03-29T10:55:04.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.947+0000 7fcaa8839640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fcaa41422b0 0x7fcaa413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:35594/0 (socket says 172.21.15.64:35594) 2024-03-29T10:55:04.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.947+0000 7fcaa8839640 1 -- 172.21.15.64:0/3046479159 learned_addr learned my addr 172.21.15.64:0/3046479159 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:04.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.948+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529254426 0 0) 0x7fcaa40aae40 con 0x7fcaa41422b0 2024-03-29T10:55:04.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.948+0000 7fcaa37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:04.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.948+0000 7fcaa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:04.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.948+0000 7fcaa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:04.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.948+0000 7fcaa37fe640 10 monclient(hunting): my global_id is 4555 2024-03-29T10:55:04.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.948+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8c003520 con 0x7fcaa41422b0 2024-03-29T10:55:04.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353485797 0 0) 0x7fcaa413fbd0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8c0048a0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1732948031 0 0) 0x7fca8c0048a0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 >> v1:172.21.15.174:6789/0 conn(0x7fcaa41422b0 legacy=0x7fcaa413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 >> v1:172.21.15.64:6789/0 conn(0x7fcaa4142690 legacy=0x7fcaa40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:04.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 monclient: found mon.noname-b 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4151940 con 0x7fcaa40b17d0 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaaaa9d640 5 monclient: authenticate success, global_id 4586 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaaaa9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fca9c0019b0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:04.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaa37fe640 10 monclient: dump: 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.949+0000 7fcaaaa9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:04.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.950+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca9c001b90 con 0x7fcaa40b17d0 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.950+0000 7fcaa37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.950+0000 7fcaa37fe640 1 -- 172.21.15.64:0/3046479159 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fca9c0019b0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.950+0000 7fcaa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.950+0000 7fcaa37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.950+0000 7fcaa37fe640 10 monclient: dump: 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.938 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.951+0000 7fcaaaa9d640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.951+0000 7fcaaaa9d640 4 set_mon_vals no callback set 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.952+0000 7fcaaaa9d640 10 monclient: shutdown 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.952+0000 7fcaaaa9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.952+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/3046479159 >> v1:172.21.15.64:6790/0 conn(0x7fcaa40b17d0 legacy=0x7fcaa40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:04.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.952+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/3046479159 shutdown_connections 2024-03-29T10:55:04.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.952+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/3046479159 >> 172.21.15.64:0/3046479159 conn(0x7fcaa4066a00 msgr2=0x7fcaa4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:04.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.953+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/3046479159 shutdown_connections 2024-03-29T10:55:04.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.955+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/3046479159 wait complete. 2024-03-29T10:55:04.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.956+0000 7fcaaaa9d640 10 monclient: build_initial_monmap 2024-03-29T10:55:04.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.956+0000 7fcaaaa9d640 10 monclient: monmap: 2024-03-29T10:55:04.943 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 1 librados: starting msgr at 2024-03-29T10:55:04.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 1 librados: starting objecter 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 1 Processor -- start 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 1 -- start start 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 1 librados: setting wanted keys 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 1 librados: calling monclient init 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.957+0000 7fcaaaa9d640 10 monclient: init 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.958+0000 7fcaaaa9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.958+0000 7fcaaaa9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.958+0000 7fcaaaa9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa40b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.958+0000 7fcaaaa9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa41422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.959+0000 7fcaaaa9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa4142690 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.959+0000 7fcaaaa9d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa406fc50 con 0x7fcaa4142690 2024-03-29T10:55:04.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.959+0000 7fcaaaa9d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa406fec0 con 0x7fcaa41422b0 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.959+0000 7fcaaaa9d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40b1190 con 0x7fcaa40b17d0 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.959+0000 7fcaaaa9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.959+0000 7fcaaaa9d640 10 monclient(hunting): authenticate will time out at 1316.257932s 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.960+0000 7fcaa3fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fcaa41422b0 0x7fcaa40713f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:39566/0 (socket says 172.21.15.64:39566) 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.960+0000 7fcaa3fff640 1 -- 172.21.15.64:0/2641046033 learned_addr learned my addr 172.21.15.64:0/2641046033 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.961+0000 7fcaa17fa640 10 client.?.objecter ms_handle_connect 0x7fcaa41422b0 2024-03-29T10:55:04.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.961+0000 7fcaa17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 10 client.?.objecter ms_handle_connect 0x7fcaa40b17d0 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565830212 0 0) 0x7fcaa40b1190 con 0x7fcaa40b17d0 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:04.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.963+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca90004eb0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.964+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532492882 0 0) 0x7fcaa406fec0 con 0x7fcaa41422b0 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.964+0000 7fcaa17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.964+0000 7fcaa17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.964+0000 7fcaa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.964+0000 7fcaa17fa640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.964+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca90006230 con 0x7fcaa41422b0 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3072142834 0 0) 0x7fca90004eb0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa406fec0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2754861975 0 0) 0x7fca90006230 con 0x7fcaa41422b0 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca90006ad0 con 0x7fcaa41422b0 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fca94003fa0 con 0x7fcaa41422b0 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 10 monclient(hunting): dump: 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:04.955 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.965+0000 7fcaa17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.966+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fca980018b0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.966+0000 7fcaa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.966+0000 7fcaa17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.966+0000 7fcaa17fa640 10 monclient(hunting): dump: 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.956 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.966+0000 7fcaa17fa640 10 client.?.objecter ms_handle_connect 0x7fcaa4142690 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.966+0000 7fcaa17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2301667517 0 0) 0x7fcaa406fec0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 >> v1:172.21.15.174:6789/0 conn(0x7fcaa41422b0 legacy=0x7fcaa40713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 >> v1:172.21.15.64:6789/0 conn(0x7fcaa4142690 legacy=0x7fcaa4071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 1 monclient: found mon.c 2024-03-29T10:55:04.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa40b1430 con 0x7fcaa40b17d0 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 10 monclient: _finish_auth 0 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca980047e0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaa2ffd640 4 set_mon_vals no callback set 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaaaa9d640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:55:04.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.967+0000 7fcaaaa9d640 10 monclient: _renew_subs 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa40b0c70 con 0x7fcaa40b17d0 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 10 monclient: _renew_subs 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa41df2f0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaaaa9d640 1 librados: init done 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fca98004af0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.968+0000 7fcaa17fa640 10 monclient: dump: 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:04.959 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fca8affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fca9801aac0 con 0x7fcaa40b17d0 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fca8affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fca8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c005080 con 0x7fcaa40b17d0 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fca98084780 con 0x7fcaa40b17d0 2024-03-29T10:55:04.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fcaa17fa640 10 client.4589.objecter ms_dispatch 0x7fcaa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:04.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fcaa17fa640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:04.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.971+0000 7fcaa17fa640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:04.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.972+0000 7fcaa17fa640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T10:55:04.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.973+0000 7fcaa3fff640 10 monclient: get_auth_request con 0x7fca9006e3c0 auth_method 0 2024-03-29T10:55:04.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.974+0000 7fcaa17fa640 10 client.4589.objecter ms_handle_connect 0x7fca9006e3c0 2024-03-29T10:55:04.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.982+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca98050060 con 0x7fcaa40b17d0 2024-03-29T10:55:04.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.982+0000 7fcaa17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:04.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:04.982+0000 7fcaa17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:05.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.312+0000 7fca8affd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:05.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.312+0000 7fca8affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:05.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.312+0000 7fca8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:05.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.312+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca7c005330 con 0x7fcaa40b17d0 2024-03-29T10:55:05.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.316+0000 7fcaa17fa640 1 -- 172.21.15.64:0/2641046033 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fca98004950 con 0x7fcaa40b17d0 2024-03-29T10:55:05.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.316+0000 7fcaa17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:05.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.316+0000 7fcaa17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:05.300 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:05.300 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:05.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.322+0000 7fca8affd640 10 librados: watch_flush enter 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.322+0000 7fca8affd640 10 librados: watch_flush exit 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.322+0000 7fca8affd640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.322+0000 7fca8affd640 10 client.4589.objecter successfully canceled tick 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.323+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 >> v1:172.21.15.174:6816/2473791532 conn(0x7fca9006e3c0 legacy=0x7fca90070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.323+0000 7fca8affd640 10 monclient: shutdown 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.323+0000 7fca8affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.323+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 >> v1:172.21.15.64:6790/0 conn(0x7fcaa40b17d0 legacy=0x7fcaa406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:05.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.324+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 shutdown_connections 2024-03-29T10:55:05.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.324+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 >> 172.21.15.64:0/2641046033 conn(0x7fcaa4066a00 msgr2=0x7fcaa40b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:05.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.327+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 shutdown_connections 2024-03-29T10:55:05.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.327+0000 7fca8affd640 1 -- 172.21.15.64:0/2641046033 wait complete. 2024-03-29T10:55:05.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:05.328+0000 7fca8affd640 1 librados: shutdown 2024-03-29T10:55:05.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:12.133 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.301+0000 7f53429e6640 10 monclient: get_monmap_and_config 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.301+0000 7f53429e6640 10 monclient: build_initial_monmap 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.301+0000 7f53429e6640 10 monclient: monmap: 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:12.302000+0000 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:12.302000+0000 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:12.282 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:12.283 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:12.283 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:12.283 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:12.283 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:12.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.302+0000 7f53429e6640 1 Processor -- start 2024-03-29T10:55:12.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.302+0000 7f53429e6640 1 -- start start 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.302+0000 7f53429e6640 10 monclient: init 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.302+0000 7f53429e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.302+0000 7f53429e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 10 monclient(hunting): picked mon.noname-c con 0x7f533c1422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 10 monclient(hunting): picked mon.noname-a con 0x7f533c142690 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 10 monclient(hunting): picked mon.noname-b con 0x7f533c0b17d0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c142c80 con 0x7f533c142690 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0aae40 con 0x7f533c1422b0 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c13fbd0 con 0x7f533c0b17d0 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:12.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.303+0000 7f53429e6640 10 monclient(hunting): authenticate will time out at 1323.601941s 2024-03-29T10:55:12.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.304+0000 7f533bfff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f533c1422b0 0x7f533c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:60272/0 (socket says 172.21.15.64:60272) 2024-03-29T10:55:12.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.304+0000 7f533bfff640 1 -- 172.21.15.64:0/3649377682 learned_addr learned my addr 172.21.15.64:0/3649377682 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:12.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.306+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190654621 0 0) 0x7f533c0aae40 con 0x7f533c1422b0 2024-03-29T10:55:12.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.306+0000 7f533affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:12.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.306+0000 7f533affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:12.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.306+0000 7f533affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:12.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.306+0000 7f533affd640 10 monclient(hunting): my global_id is 4561 2024-03-29T10:55:12.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.306+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324003520 con 0x7f533c1422b0 2024-03-29T10:55:12.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.307+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895237111 0 0) 0x7f5324003520 con 0x7f533c1422b0 2024-03-29T10:55:12.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.307+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 >> v1:172.21.15.64:6790/0 conn(0x7f533c0b17d0 legacy=0x7f533c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:12.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.307+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 >> v1:172.21.15.64:6789/0 conn(0x7f533c142690 legacy=0x7f533c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:12.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:12.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 1 monclient: found mon.noname-c 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c151940 con 0x7f533c1422b0 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: _finish_auth 0 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f53429e6640 5 monclient: authenticate success, global_id 4561 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f53429e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5330001690 con 0x7f533c1422b0 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:12.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: dump: 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f53429e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:12.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 1 -- 172.21.15.64:0/3649377682 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53300018b0 con 0x7f533c1422b0 2024-03-29T10:55:12.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f533affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:12.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f53429e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:12.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.309+0000 7f53429e6640 4 set_mon_vals no callback set 2024-03-29T10:55:12.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 10 monclient: shutdown 2024-03-29T10:55:12.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 1 -- 172.21.15.64:0/3649377682 >> v1:172.21.15.174:6789/0 conn(0x7f533c1422b0 legacy=0x7f533c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 1 -- 172.21.15.64:0/3649377682 shutdown_connections 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 1 -- 172.21.15.64:0/3649377682 >> 172.21.15.64:0/3649377682 conn(0x7f533c066a00 msgr2=0x7f533c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 1 -- 172.21.15.64:0/3649377682 shutdown_connections 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.310+0000 7f53429e6640 1 -- 172.21.15.64:0/3649377682 wait complete. 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.311+0000 7f53429e6640 10 monclient: build_initial_monmap 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.311+0000 7f53429e6640 10 monclient: monmap: 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:12.296 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 1 librados: starting msgr at 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 1 librados: starting objecter 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 1 Processor -- start 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 1 -- start start 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 1 librados: setting wanted keys 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 1 librados: calling monclient init 2024-03-29T10:55:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.312+0000 7f53429e6640 10 monclient: init 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient(hunting): picked mon.noname-a con 0x7f533c0b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient(hunting): picked mon.noname-c con 0x7f533c1422b0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient(hunting): picked mon.noname-b con 0x7f533c142690 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0ac980 con 0x7f533c0b17d0 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c1db640 con 0x7f533c142690 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0b6f90 con 0x7f533c1422b0 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.313+0000 7f53429e6640 10 monclient(hunting): authenticate will time out at 1323.612621s 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.316+0000 7f5340f83640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f533c142690 0x7f533c1daf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:35980/0 (socket says 172.21.15.64:35980) 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.316+0000 7f5340f83640 1 -- 172.21.15.64:0/1256478554 learned_addr learned my addr 172.21.15.64:0/1256478554 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 client.?.objecter ms_handle_connect 0x7f533c142690 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 client.?.objecter ms_handle_connect 0x7f533c0b17d0 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287359292 0 0) 0x7f533c1db640 con 0x7f533c142690 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:12.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 10 monclient(hunting): my global_id is 4564 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5328004eb0 con 0x7f533c142690 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.317+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240642321 0 0) 0x7f533c0ac980 con 0x7f533c0b17d0 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c1db640 con 0x7f533c0b17d0 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 client.?.objecter ms_handle_connect 0x7f533c1422b0 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148329958 0 0) 0x7f533c0b6f90 con 0x7f533c1422b0 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:12.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.318+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c0ac980 con 0x7f533c1422b0 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689792851 0 0) 0x7f533c1db640 con 0x7f533c0b17d0 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f533c0b6f90 con 0x7f533c0b17d0 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5330004940 con 0x7f533c0b17d0 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:12.304 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678367814 0 0) 0x7f533c0ac980 con 0x7f533c1422b0 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f533c1db640 con 0x7f533c1422b0 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3915631669 0 0) 0x7f533c0b6f90 con 0x7f533c0b17d0 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 >> v1:172.21.15.64:6790/0 conn(0x7f533c1422b0 legacy=0x7f533c1da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:12.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 >> v1:172.21.15.174:6789/0 conn(0x7f533c142690 legacy=0x7f533c1daf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 monclient: found mon.a 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c150560 con 0x7f533c0b17d0 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.319+0000 7f5338ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:12.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 5 monclient: authenticate success, global_id 4530 2024-03-29T10:55:12.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 10 monclient: _renew_subs 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c150810 con 0x7f533c0b17d0 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 10 monclient: _renew_subs 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c150e10 con 0x7f533c0b17d0 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.320+0000 7f53429e6640 1 librados: init done 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f53429e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5330001aa0 con 0x7f533c0b17d0 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f53429e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f53429e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c075f10 con 0x7f533c0b17d0 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f5338ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5330004f30 con 0x7f533c0b17d0 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f5338ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f5338ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f5338ff9640 10 monclient: dump: 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:12.309 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.322+0000 7f533a7fc640 4 set_mon_vals no callback set 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.325+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f533001af00 con 0x7f533c0b17d0 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.325+0000 7f533b7fe640 10 monclient: get_auth_request con 0x7f532806e060 auth_method 0 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.325+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f5330084ab0 con 0x7f533c0b17d0 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.326+0000 7f5338ff9640 10 client.4530.objecter ms_dispatch 0x7f533c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.326+0000 7f5338ff9640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.326+0000 7f5338ff9640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.326+0000 7f5338ff9640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T10:55:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.326+0000 7f5338ff9640 10 client.4530.objecter ms_handle_connect 0x7f532806e060 2024-03-29T10:55:12.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.331+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5330050390 con 0x7f533c0b17d0 2024-03-29T10:55:12.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.331+0000 7f5338ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:12.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.331+0000 7f5338ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:12.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.662+0000 7f53429e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:12.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.662+0000 7f53429e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:12.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.662+0000 7f53429e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:12.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.662+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f533c0b2a30 con 0x7f533c0b17d0 2024-03-29T10:55:12.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.664+0000 7f5338ff9640 1 -- 172.21.15.64:0/1256478554 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f53300522c0 con 0x7f533c0b17d0 2024-03-29T10:55:12.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.664+0000 7f5338ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:12.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.664+0000 7f5338ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:12.646 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:12.646 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:12.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 10 librados: watch_flush enter 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 10 librados: watch_flush exit 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 10 client.4530.objecter successfully canceled tick 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 >> v1:172.21.15.174:6816/2473791532 conn(0x7f532806e060 legacy=0x7f5328070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 10 monclient: shutdown 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:12.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.668+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 >> v1:172.21.15.64:6789/0 conn(0x7f533c0b17d0 legacy=0x7f533c0b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:12.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.671+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 shutdown_connections 2024-03-29T10:55:12.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.671+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 >> 172.21.15.64:0/1256478554 conn(0x7f533c066a00 msgr2=0x7f533c13c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:12.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.671+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 shutdown_connections 2024-03-29T10:55:12.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.671+0000 7f53429e6640 1 -- 172.21.15.64:0/1256478554 wait complete. 2024-03-29T10:55:12.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:12.672+0000 7f53429e6640 1 librados: shutdown 2024-03-29T10:55:12.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:19.477 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:19.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.643+0000 7fc298f74640 10 monclient: get_monmap_and_config 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.643+0000 7fc298f74640 10 monclient: build_initial_monmap 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.643+0000 7fc298f74640 10 monclient: monmap: 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:19.644215+0000 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:19.644215+0000 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:19.625 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:19.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.647+0000 7fc298f74640 1 Processor -- start 2024-03-29T10:55:19.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.647+0000 7fc298f74640 1 -- start start 2024-03-29T10:55:19.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.647+0000 7fc298f74640 10 monclient: init 2024-03-29T10:55:19.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c034850 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c0c9550 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c02aac0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c02c440 con 0x7fc28c034850 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c02aed0 con 0x7fc28c02aac0 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c02b190 con 0x7fc28c0c9550 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:19.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.649+0000 7fc298f74640 10 monclient(hunting): authenticate will time out at 1330.948624s 2024-03-29T10:55:19.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.654+0000 7fc292ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc28c0c9550 0x7fc28c0cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46381/0 (socket says 172.21.15.64:46381) 2024-03-29T10:55:19.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.654+0000 7fc292ffd640 1 -- 172.21.15.64:0/3999281448 learned_addr learned my addr 172.21.15.64:0/3999281448 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:19.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.655+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073531425 0 0) 0x7fc28c02b190 con 0x7fc28c0c9550 2024-03-29T10:55:19.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.655+0000 7fc2927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:19.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.655+0000 7fc2927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:19.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.655+0000 7fc2927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:19.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.655+0000 7fc2927fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.655+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc284003520 con 0x7fc28c0c9550 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738181513 0 0) 0x7fc284003520 con 0x7fc28c0c9550 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 >> v1:172.21.15.174:6789/0 conn(0x7fc28c02aac0 legacy=0x7fc28c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 >> v1:172.21.15.64:6789/0 conn(0x7fc28c034850 legacy=0x7fc28c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c033060 con 0x7fc28c0c9550 2024-03-29T10:55:19.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc288001690 con 0x7fc28c0c9550 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc2927fc640 10 monclient: dump: 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:19.638 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:19.639 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:19.639 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:19.639 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:19.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc298f74640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:55:19.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.656+0000 7fc298f74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc288001860 con 0x7fc28c0c9550 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc2927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc2927fc640 1 -- 172.21.15.64:0/3999281448 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc288001cb0 con 0x7fc28c0c9550 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc2927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc2927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc2927fc640 10 monclient: dump: 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:19.640 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc298f74640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.657+0000 7fc298f74640 4 set_mon_vals no callback set 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 10 monclient: shutdown 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 1 -- 172.21.15.64:0/3999281448 >> v1:172.21.15.64:6790/0 conn(0x7fc28c0c9550 legacy=0x7fc28c0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 1 -- 172.21.15.64:0/3999281448 shutdown_connections 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 1 -- 172.21.15.64:0/3999281448 >> 172.21.15.64:0/3999281448 conn(0x7fc28c0058f0 msgr2=0x7fc28c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 1 -- 172.21.15.64:0/3999281448 shutdown_connections 2024-03-29T10:55:19.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.658+0000 7fc298f74640 1 -- 172.21.15.64:0/3999281448 wait complete. 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.659+0000 7fc298f74640 10 monclient: build_initial_monmap 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.659+0000 7fc298f74640 10 monclient: monmap: 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:19.642 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.660+0000 7fc298f74640 1 librados: starting msgr at 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.660+0000 7fc298f74640 1 librados: starting objecter 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.660+0000 7fc298f74640 1 Processor -- start 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.662+0000 7fc298f74640 1 -- start start 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.662+0000 7fc298f74640 1 librados: setting wanted keys 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.662+0000 7fc298f74640 1 librados: calling monclient init 2024-03-29T10:55:19.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.662+0000 7fc298f74640 10 monclient: init 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c02aac0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c034850 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c0c9550 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c160b90 con 0x7fc28c0c9550 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c15e7f0 con 0x7fc28c034850 2024-03-29T10:55:19.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c161a60 con 0x7fc28c02aac0 2024-03-29T10:55:19.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:19.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.663+0000 7fc298f74640 10 monclient(hunting): authenticate will time out at 1330.962472s 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc292ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc28c034850 0x7fc28c161350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:59671/0 (socket says 172.21.15.64:59671) 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc293fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fc28c0c9550 0x7fc28c031a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55109/0 (socket says 172.21.15.64:55109) 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc2937fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc28c02aac0 0x7fc28c0be770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46477/0 (socket says 172.21.15.64:46477) 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc293fff640 1 -- 172.21.15.64:0/3812689228 learned_addr learned my addr 172.21.15.64:0/3812689228 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 client.?.objecter ms_handle_connect 0x7fc28c02aac0 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 client.?.objecter ms_handle_connect 0x7fc28c0c9550 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622146148 0 0) 0x7fc28c160b90 con 0x7fc28c0c9550 2024-03-29T10:55:19.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc278004eb0 con 0x7fc28c0c9550 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 client.?.objecter ms_handle_connect 0x7fc28c034850 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375133374 0 0) 0x7fc28c15e7f0 con 0x7fc28c034850 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:55:19.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28c160b90 con 0x7fc28c034850 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.670+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4096408748 0 0) 0x7fc278004eb0 con 0x7fc28c0c9550 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc28c15e7f0 con 0x7fc28c0c9550 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc29404fdd0 con 0x7fc28c0c9550 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient(hunting): dump: 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.659 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2531372798 0 0) 0x7fc28c15e7f0 con 0x7fc28c0c9550 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 >> v1:172.21.15.64:6790/0 conn(0x7fc28c02aac0 legacy=0x7fc28c0be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 >> v1:172.21.15.174:6789/0 conn(0x7fc28c034850 legacy=0x7fc28c161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 monclient: found mon.a 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:19.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c02e1d0 con 0x7fc28c0c9550 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient: _finish_auth 0 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc27ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 5 monclient: authenticate success, global_id 4536 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 10 monclient: _renew_subs 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c160e30 con 0x7fc28c0c9550 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 10 monclient: _renew_subs 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c02ca70 con 0x7fc28c0c9550 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.671+0000 7fc298f74640 1 librados: init done 2024-03-29T10:55:19.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.672+0000 7fc298f74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.672+0000 7fc298f74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.672+0000 7fc298f74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.673+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc29404f400 con 0x7fc28c0c9550 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.672+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c161070 con 0x7fc28c0c9550 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.673+0000 7fc27ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.673+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fc294075710 con 0x7fc28c0c9550 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.673+0000 7fc27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.673+0000 7fc27ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.673+0000 7fc27ffff640 10 monclient: dump: 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.662 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fc294067bf0 con 0x7fc28c0c9550 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7fc2940ff440 con 0x7fc28c0c9550 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc27ffff640 10 client.4536.objecter ms_dispatch 0x7fc294000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc27ffff640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc27ffff640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc27ffff640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.674+0000 7fc2917fa640 4 set_mon_vals no callback set 2024-03-29T10:55:19.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.675+0000 7fc2937fe640 10 monclient: get_auth_request con 0x7fc27806dc20 auth_method 0 2024-03-29T10:55:19.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.682+0000 7fc27ffff640 10 client.4536.objecter ms_handle_connect 0x7fc27806dc20 2024-03-29T10:55:19.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.683+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc28c161070 con 0x7fc28c0c9550 2024-03-29T10:55:19.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.683+0000 7fc27ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:19.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:19.683+0000 7fc27ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:19.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.008+0000 7fc298f74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:19.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.008+0000 7fc298f74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:19.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.008+0000 7fc298f74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:55:19.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.008+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc28c161070 con 0x7fc28c0c9550 2024-03-29T10:55:19.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.010+0000 7fc27ffff640 1 -- 172.21.15.64:0/3812689228 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7fc28c161070 con 0x7fc28c0c9550 2024-03-29T10:55:19.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.010+0000 7fc27ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:19.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.010+0000 7fc27ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:19.993 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:19.993 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:19.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.016+0000 7fc298f74640 10 librados: watch_flush enter 2024-03-29T10:55:19.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.016+0000 7fc298f74640 10 librados: watch_flush exit 2024-03-29T10:55:19.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.016+0000 7fc298f74640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T10:55:19.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.016+0000 7fc298f74640 10 client.4536.objecter successfully canceled tick 2024-03-29T10:55:19.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.018+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 >> v1:172.21.15.174:6816/2473791532 conn(0x7fc27806dc20 legacy=0x7fc2780700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:19.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.018+0000 7fc298f74640 10 monclient: shutdown 2024-03-29T10:55:19.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.018+0000 7fc298f74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:20.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.018+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 >> v1:172.21.15.64:6789/0 conn(0x7fc28c0c9550 legacy=0x7fc28c031a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:20.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.018+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 shutdown_connections 2024-03-29T10:55:20.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.018+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 >> 172.21.15.64:0/3812689228 conn(0x7fc28c0058f0 msgr2=0x7fc28c0c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:20.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.019+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 shutdown_connections 2024-03-29T10:55:20.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.019+0000 7fc298f74640 1 -- 172.21.15.64:0/3812689228 wait complete. 2024-03-29T10:55:20.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:20.019+0000 7fc298f74640 1 librados: shutdown 2024-03-29T10:55:20.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:26.824 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:26.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.980+0000 7f001e006640 10 monclient: get_monmap_and_config 2024-03-29T10:55:26.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.980+0000 7f001e006640 10 monclient: build_initial_monmap 2024-03-29T10:55:26.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.980+0000 7f001e006640 10 monclient: monmap: 2024-03-29T10:55:26.968 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:26.968 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:26.968 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:26.981270+0000 2024-03-29T10:55:26.969 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:26.981270+0000 2024-03-29T10:55:26.969 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:26.969 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:26.969 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:26.969 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:26.969 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.981+0000 7f001e006640 1 Processor -- start 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.981+0000 7f001e006640 1 -- start start 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.981+0000 7f001e006640 10 monclient: init 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient(hunting): picked mon.noname-b con 0x7f00180ad6c0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient(hunting): picked mon.noname-c con 0x7f00180af6c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient(hunting): picked mon.noname-a con 0x7f001814fdc0 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180a94e0 con 0x7f001814fdc0 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180af3a0 con 0x7f00180af6c0 2024-03-29T10:55:26.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180afb60 con 0x7f00180ad6c0 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f001e006640 10 monclient(hunting): authenticate will time out at 1338.281247s 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f00177fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f00180ad6c0 0x7f001814f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41474/0 (socket says 172.21.15.64:41474) 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.982+0000 7f00177fe640 1 -- 172.21.15.64:0/3513230068 learned_addr learned my addr 172.21.15.64:0/3513230068 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009031559 0 0) 0x7f00180afb60 con 0x7f00180ad6c0 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0000003520 con 0x7f00180ad6c0 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644916317 0 0) 0x7f00180a94e0 con 0x7f001814fdc0 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:26.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00180afb60 con 0x7f001814fdc0 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276206072 0 0) 0x7f0000003520 con 0x7f00180ad6c0 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.983+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 >> v1:172.21.15.174:6789/0 conn(0x7f00180af6c0 legacy=0x7f0018152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 >> v1:172.21.15.64:6789/0 conn(0x7f001814fdc0 legacy=0x7f001815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180aab80 con 0x7f00180ad6c0 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:26.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0008001610 con 0x7f00180ad6c0 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.984+0000 7f00167fc640 10 monclient: dump: 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:26.973 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f001e006640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f001e006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0008001770 con 0x7f00180ad6c0 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f00167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f00167fc640 1 -- 172.21.15.64:0/3513230068 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0008001b80 con 0x7f00180ad6c0 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f00167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f00167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f00167fc640 10 monclient: dump: 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:26.974 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:26.975 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:26.975 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:26.975 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:26.975 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:26.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f001e006640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:26.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.985+0000 7f001e006640 4 set_mon_vals no callback set 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.986+0000 7f001e006640 10 monclient: shutdown 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.986+0000 7f001e006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.986+0000 7f001e006640 1 -- 172.21.15.64:0/3513230068 >> v1:172.21.15.64:6790/0 conn(0x7f00180ad6c0 legacy=0x7f001814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.987+0000 7f001e006640 1 -- 172.21.15.64:0/3513230068 shutdown_connections 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.987+0000 7f001e006640 1 -- 172.21.15.64:0/3513230068 >> 172.21.15.64:0/3513230068 conn(0x7f00180b35e0 msgr2=0x7f00180b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.987+0000 7f001e006640 1 -- 172.21.15.64:0/3513230068 shutdown_connections 2024-03-29T10:55:26.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.987+0000 7f001e006640 1 -- 172.21.15.64:0/3513230068 wait complete. 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.988+0000 7f001e006640 10 monclient: build_initial_monmap 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.988+0000 7f001e006640 10 monclient: monmap: 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:26.982 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.988+0000 7f001e006640 1 librados: starting msgr at 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.988+0000 7f001e006640 1 librados: starting objecter 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.989+0000 7f001e006640 1 Processor -- start 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.989+0000 7f001e006640 1 -- start start 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.989+0000 7f001e006640 1 librados: setting wanted keys 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.989+0000 7f001e006640 1 librados: calling monclient init 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.989+0000 7f001e006640 10 monclient: init 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient(hunting): picked mon.noname-b con 0x7f00180ad6c0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient(hunting): picked mon.noname-a con 0x7f00180af6c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient(hunting): picked mon.noname-c con 0x7f001814fdc0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:26.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180716c0 con 0x7f00180af6c0 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180705f0 con 0x7f00180ad6c0 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0018151c00 con 0x7f001814fdc0 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f001e006640 10 monclient(hunting): authenticate will time out at 1338.289373s 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f0017fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f001814fdc0 0x7f00181eeaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:43620/0 (socket says 172.21.15.64:43620) 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.990+0000 7f0017fff640 1 -- 172.21.15.64:0/1976469382 learned_addr learned my addr 172.21.15.64:0/1976469382 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 client.?.objecter ms_handle_connect 0x7f001814fdc0 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030437180 0 0) 0x7f0018151c00 con 0x7f001814fdc0 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004004eb0 con 0x7f001814fdc0 2024-03-29T10:55:26.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4019994263 0 0) 0x7f0004004eb0 con 0x7f001814fdc0 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0018151c00 con 0x7f001814fdc0 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0010001690 con 0x7f001814fdc0 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): dump: 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:26.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 660792326 0 0) 0x7f0018151c00 con 0x7f001814fdc0 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.991+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 >> v1:172.21.15.174:6789/0 conn(0x7f00180ad6c0 legacy=0x7f0018070e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.992+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 >> v1:172.21.15.64:6789/0 conn(0x7f00180af6c0 legacy=0x7f00181e4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.992+0000 7efffffff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.992+0000 7efffffff640 1 monclient: found mon.c 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.992+0000 7efffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.992+0000 7efffffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.992+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018151d90 con 0x7f001814fdc0 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7efffffff640 10 monclient: _finish_auth 0 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7efffffff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7efffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7efffffff640 10 client.?.objecter ms_handle_connect 0x7f00180af6c0 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7efffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:26.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 5 monclient: authenticate success, global_id 4601 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 10 monclient: _renew_subs 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00181e4780 con 0x7f001814fdc0 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 10 monclient: _renew_subs 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00181f0210 con 0x7f001814fdc0 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.993+0000 7f001e006640 1 librados: init done 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0010001970 con 0x7f001814fdc0 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7efffffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7f0015ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f0010003dc0 con 0x7f001814fdc0 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7efffffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7efffffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:26.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7efffffff640 10 monclient: dump: 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.994+0000 7f001e006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.995+0000 7f001e006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.995+0000 7f001e006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.995+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f0010003f20 con 0x7f001814fdc0 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.995+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff8005080 con 0x7f001814fdc0 2024-03-29T10:55:26.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.996+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f0010084ef0 con 0x7f001814fdc0 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.996+0000 7efffffff640 10 client.4601.objecter ms_dispatch 0x7f0018000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.996+0000 7efffffff640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.996+0000 7efffffff640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.996+0000 7efffffff640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.996+0000 7f00177fe640 10 monclient: get_auth_request con 0x7f000406dc40 auth_method 0 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:26.997+0000 7efffffff640 10 client.4601.objecter ms_handle_connect 0x7f000406dc40 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.002+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00100507d0 con 0x7f001814fdc0 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.002+0000 7efffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:26.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.002+0000 7efffffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:27.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.329+0000 7f001e006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:27.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.329+0000 7f001e006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:27.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.329+0000 7f001e006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:27.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.329+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efff8005330 con 0x7f001814fdc0 2024-03-29T10:55:27.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.330+0000 7efffffff640 1 -- 172.21.15.64:0/1976469382 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f0010052700 con 0x7f001814fdc0 2024-03-29T10:55:27.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.330+0000 7efffffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:27.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.330+0000 7efffffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:27.312 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:27.312 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 10 librados: watch_flush enter 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 10 librados: watch_flush exit 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 10 client.4601.objecter successfully canceled tick 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 >> v1:172.21.15.174:6816/2473791532 conn(0x7f000406dc40 legacy=0x7f0004070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 10 monclient: shutdown 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.334+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 >> v1:172.21.15.64:6790/0 conn(0x7f001814fdc0 legacy=0x7f00181eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.335+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 shutdown_connections 2024-03-29T10:55:27.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.335+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 >> 172.21.15.64:0/1976469382 conn(0x7f00180b35e0 msgr2=0x7f00180ac070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:27.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.335+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 shutdown_connections 2024-03-29T10:55:27.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.335+0000 7f001e006640 1 -- 172.21.15.64:0/1976469382 wait complete. 2024-03-29T10:55:27.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:27.335+0000 7f001e006640 1 librados: shutdown 2024-03-29T10:55:27.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:34.140 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:34.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.325+0000 7f1647a45640 10 monclient: get_monmap_and_config 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.325+0000 7f1647a45640 10 monclient: build_initial_monmap 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.326+0000 7f1647a45640 10 monclient: monmap: 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:34.326840+0000 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:34.326840+0000 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:34.307 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.327+0000 7f1647a45640 1 Processor -- start 2024-03-29T10:55:34.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.327+0000 7f1647a45640 1 -- start start 2024-03-29T10:55:34.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.327+0000 7f1647a45640 10 monclient: init 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient(hunting): picked mon.noname-c con 0x7f16400b0770 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient(hunting): picked mon.noname-a con 0x7f164014a700 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient(hunting): picked mon.noname-b con 0x7f16400ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16400aba70 con 0x7f164014a700 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16400aecb0 con 0x7f16400b0770 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16400af150 con 0x7f16400ac5a0 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:34.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1647a45640 10 monclient(hunting): authenticate will time out at 1345.627254s 2024-03-29T10:55:34.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1645fe2640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f16400ac5a0 0x7f1640151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33948/0 (socket says 172.21.15.64:33948) 2024-03-29T10:55:34.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.328+0000 7f1645fe2640 1 -- 172.21.15.64:0/1870422059 learned_addr learned my addr 172.21.15.64:0/1870422059 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478190776 0 0) 0x7f16400af150 con 0x7f16400ac5a0 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c003520 con 0x7f16400ac5a0 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890104316 0 0) 0x7f16400aecb0 con 0x7f16400b0770 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 10 monclient(hunting): my global_id is 4570 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.329+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16400af150 con 0x7f16400b0770 2024-03-29T10:55:34.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927794606 0 0) 0x7f16400af150 con 0x7f16400b0770 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 >> v1:172.21.15.64:6790/0 conn(0x7f16400ac5a0 legacy=0x7f1640151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 >> v1:172.21.15.64:6789/0 conn(0x7f164014a700 legacy=0x7f16400abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 1 monclient: found mon.noname-c 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16400ac9d0 con 0x7f16400b0770 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f163c0015f0 con 0x7f16400b0770 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1647a45640 5 monclient: authenticate success, global_id 4570 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1647a45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:34.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1637fff640 10 monclient: dump: 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.330+0000 7f1647a45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f163c001750 con 0x7f16400b0770 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1637fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1637fff640 1 -- 172.21.15.64:0/1870422059 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f163c001ba0 con 0x7f16400b0770 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1637fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1637fff640 10 monclient: dump: 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.315 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1647a45640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.331+0000 7f1647a45640 4 set_mon_vals no callback set 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.332+0000 7f1647a45640 10 monclient: shutdown 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.332+0000 7f1647a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.332+0000 7f1647a45640 1 -- 172.21.15.64:0/1870422059 >> v1:172.21.15.174:6789/0 conn(0x7f16400b0770 legacy=0x7f1640149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.332+0000 7f1647a45640 1 -- 172.21.15.64:0/1870422059 shutdown_connections 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.332+0000 7f1647a45640 1 -- 172.21.15.64:0/1870422059 >> 172.21.15.64:0/1870422059 conn(0x7f164006b660 msgr2=0x7f164006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.333+0000 7f1647a45640 1 -- 172.21.15.64:0/1870422059 shutdown_connections 2024-03-29T10:55:34.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.333+0000 7f1647a45640 1 -- 172.21.15.64:0/1870422059 wait complete. 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.333+0000 7f1647a45640 10 monclient: build_initial_monmap 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.333+0000 7f1647a45640 10 monclient: monmap: 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 1 librados: starting msgr at 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 1 librados: starting objecter 2024-03-29T10:55:34.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 1 Processor -- start 2024-03-29T10:55:34.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 1 -- start start 2024-03-29T10:55:34.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 1 librados: setting wanted keys 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 1 librados: calling monclient init 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.334+0000 7f1647a45640 10 monclient: init 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient(hunting): picked mon.noname-c con 0x7f16400ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient(hunting): picked mon.noname-a con 0x7f16400b0770 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient(hunting): picked mon.noname-b con 0x7f164014a700 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640145330 con 0x7f16400b0770 2024-03-29T10:55:34.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16401454c0 con 0x7f164014a700 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640145650 con 0x7f16400ac5a0 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1647a45640 10 monclient(hunting): authenticate will time out at 1345.634423s 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1645fe2640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f164014a700 0x7f1640126fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:51794/0 (socket says 172.21.15.64:51794) 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1645fe2640 1 -- 172.21.15.64:0/2142674405 learned_addr learned my addr 172.21.15.64:0/2142674405 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1635ffb640 10 client.?.objecter ms_handle_connect 0x7f16400b0770 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1635ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1635ffb640 10 client.?.objecter ms_handle_connect 0x7f164014a700 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.335+0000 7f1635ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 client.?.objecter ms_handle_connect 0x7f16400ac5a0 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:34.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001185720 0 0) 0x7f16400ac9d0 con 0x7f16400ac5a0 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1624004eb0 con 0x7f16400ac5a0 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602609233 0 0) 0x7f16401454c0 con 0x7f164014a700 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 10 monclient(hunting): my global_id is 4573 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.336+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16400ac9d0 con 0x7f164014a700 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020817733 0 0) 0x7f1640145330 con 0x7f16400b0770 2024-03-29T10:55:34.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16401454c0 con 0x7f16400b0770 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1829309663 0 0) 0x7f16400ac9d0 con 0x7f164014a700 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1640145330 con 0x7f164014a700 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1638001c50 con 0x7f164014a700 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.323 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 416073117 0 0) 0x7f1624004eb0 con 0x7f16400ac5a0 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16400ac9d0 con 0x7f16400ac5a0 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f163c0018c0 con 0x7f16400ac5a0 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:34.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1704333764 0 0) 0x7f1640145330 con 0x7f164014a700 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 >> v1:172.21.15.64:6790/0 conn(0x7f16400ac5a0 legacy=0x7f16401df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.337+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 >> v1:172.21.15.64:6789/0 conn(0x7f16400b0770 legacy=0x7f16401268a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 1 monclient: found mon.b 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16401dfa20 con 0x7f164014a700 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 5 monclient: authenticate success, global_id 4573 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 10 monclient: _renew_subs 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16401dfc70 con 0x7f164014a700 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 10 monclient: _renew_subs 2024-03-29T10:55:34.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16401e77d0 con 0x7f164014a700 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1647a45640 1 librados: init done 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1638002080 con 0x7f164014a700 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1638004dc0 con 0x7f164014a700 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f1635ffb640 10 monclient: dump: 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:34.327 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:34.328 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:34.328 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:34.328 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:34.328 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:34.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.338+0000 7f16377fe640 4 set_mon_vals no callback set 2024-03-29T10:55:34.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.339+0000 7f1647a45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:34.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.340+0000 7f1647a45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:34.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.340+0000 7f1647a45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:34.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.340+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1620005080 con 0x7f164014a700 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.341+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f1638004f20 con 0x7f164014a700 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.342+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f1638084e00 con 0x7f164014a700 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.342+0000 7f1635ffb640 10 client.4573.objecter ms_dispatch 0x7f1640000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.342+0000 7f1635ffb640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.342+0000 7f1635ffb640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.342+0000 7f1635ffb640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.342+0000 7f16457e1640 10 monclient: get_auth_request con 0x7f162406def0 auth_method 0 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.343+0000 7f1635ffb640 10 client.4573.objecter ms_handle_connect 0x7f162406def0 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.347+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1638050510 con 0x7f164014a700 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.348+0000 7f1635ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:34.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.348+0000 7f1635ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:34.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.768+0000 7f1647a45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:34.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.768+0000 7f1647a45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:34.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.768+0000 7f1647a45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:34.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.768+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1620005330 con 0x7f164014a700 2024-03-29T10:55:34.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.769+0000 7f1635ffb640 1 -- 172.21.15.64:0/2142674405 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f1638052610 con 0x7f164014a700 2024-03-29T10:55:34.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.769+0000 7f1635ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:34.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.769+0000 7f1635ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:34.750 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:34.750 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:34.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 10 librados: watch_flush enter 2024-03-29T10:55:34.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 10 librados: watch_flush exit 2024-03-29T10:55:34.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T10:55:34.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 10 client.4573.objecter successfully canceled tick 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 >> v1:172.21.15.174:6816/2473791532 conn(0x7f162406def0 legacy=0x7f16240703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 10 monclient: shutdown 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 >> v1:172.21.15.174:6789/0 conn(0x7f164014a700 legacy=0x7f1640126fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 shutdown_connections 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.773+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 >> 172.21.15.64:0/2142674405 conn(0x7f164006b660 msgr2=0x7f16400b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.774+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 shutdown_connections 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.774+0000 7f1647a45640 1 -- 172.21.15.64:0/2142674405 wait complete. 2024-03-29T10:55:34.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:34.774+0000 7f1647a45640 1 librados: shutdown 2024-03-29T10:55:34.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:41.578 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:41.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.749+0000 7f4b08a2b640 10 monclient: get_monmap_and_config 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.749+0000 7f4b08a2b640 10 monclient: build_initial_monmap 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.749+0000 7f4b08a2b640 10 monclient: monmap: 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:41.750907+0000 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:41.750907+0000 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:41.731 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:41.732 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:41.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.750+0000 7f4b08a2b640 1 Processor -- start 2024-03-29T10:55:41.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.750+0000 7f4b08a2b640 1 -- start start 2024-03-29T10:55:41.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.750+0000 7f4b08a2b640 10 monclient: init 2024-03-29T10:55:41.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.751+0000 7f4b08a2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:41.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.751+0000 7f4b08a2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:41.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.751+0000 7f4b08a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f4afc038c60 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:41.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4afc0324b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:41.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f4afc0d11e0 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:41.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc0caf80 con 0x7f4afc0d11e0 2024-03-29T10:55:41.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc039850 con 0x7f4afc0324b0 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc0c78d0 con 0x7f4afc038c60 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.754+0000 7f4b08a2b640 10 monclient(hunting): authenticate will time out at 1353.054721s 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b037fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4afc038c60 0x7f4afc0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56369/0 (socket says 172.21.15.64:56369) 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b037fe640 1 -- 172.21.15.64:0/302455916 learned_addr learned my addr 172.21.15.64:0/302455916 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426016036 0 0) 0x7f4afc0c78d0 con 0x7f4afc038c60 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b027fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.755+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af4003520 con 0x7f4afc038c60 2024-03-29T10:55:41.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.756+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70790676 0 0) 0x7f4af4003520 con 0x7f4afc038c60 2024-03-29T10:55:41.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.756+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 >> v1:172.21.15.174:6789/0 conn(0x7f4afc0324b0 legacy=0x7f4afc0cb4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:41.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.757+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 >> v1:172.21.15.64:6789/0 conn(0x7f4afc0d11e0 legacy=0x7f4afc0d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:41.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:41.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:41.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc0c7a60 con 0x7f4afc038c60 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4af8001670 con 0x7f4afc038c60 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.758+0000 7f4b027fc640 10 monclient: dump: 2024-03-29T10:55:41.742 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:41.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.759+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4af80018d0 con 0x7f4afc038c60 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.759+0000 7f4b027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.759+0000 7f4b027fc640 1 -- 172.21.15.64:0/302455916 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4af8001d20 con 0x7f4afc038c60 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.759+0000 7f4b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.759+0000 7f4b027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.759+0000 7f4b027fc640 10 monclient: dump: 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:41.744 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:41.745 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:41.745 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:41.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.760+0000 7f4b08a2b640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:55:41.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.760+0000 7f4b08a2b640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:41.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.760+0000 7f4b08a2b640 4 set_mon_vals no callback set 2024-03-29T10:55:41.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.761+0000 7f4b08a2b640 10 monclient: shutdown 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.761+0000 7f4b08a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.761+0000 7f4b08a2b640 1 -- 172.21.15.64:0/302455916 >> v1:172.21.15.64:6790/0 conn(0x7f4afc038c60 legacy=0x7f4afc0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.761+0000 7f4b08a2b640 1 -- 172.21.15.64:0/302455916 shutdown_connections 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.761+0000 7f4b08a2b640 1 -- 172.21.15.64:0/302455916 >> 172.21.15.64:0/302455916 conn(0x7f4afc005c20 msgr2=0x7f4afc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.761+0000 7f4b08a2b640 1 -- 172.21.15.64:0/302455916 shutdown_connections 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.762+0000 7f4b08a2b640 1 -- 172.21.15.64:0/302455916 wait complete. 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.763+0000 7f4b08a2b640 10 monclient: build_initial_monmap 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.763+0000 7f4b08a2b640 10 monclient: monmap: 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:41.747 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 1 librados: starting msgr at 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 1 librados: starting objecter 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 1 Processor -- start 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 1 -- start start 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 1 librados: setting wanted keys 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 1 librados: calling monclient init 2024-03-29T10:55:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.764+0000 7f4b08a2b640 10 monclient: init 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.765+0000 7f4b08a2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.766+0000 7f4b08a2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f4afc0324b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4afc038c60 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f4afc0d11e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc036860 con 0x7f4afc0324b0 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc036b40 con 0x7f4afc0d11e0 2024-03-29T10:55:41.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc034980 con 0x7f4afc038c60 2024-03-29T10:55:41.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:41.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.767+0000 7f4b08a2b640 10 monclient(hunting): authenticate will time out at 1353.067149s 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.772+0000 7f4b03fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f4afc0d11e0 0x7f4afc0c9460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:55339/0 (socket says 172.21.15.64:55339) 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.772+0000 7f4b03fff640 1 -- 172.21.15.64:0/2798000868 learned_addr learned my addr 172.21.15.64:0/2798000868 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 10 client.?.objecter ms_handle_connect 0x7f4afc0d11e0 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711081723 0 0) 0x7f4afc036b40 con 0x7f4afc0d11e0 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:41.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 10 monclient(hunting): my global_id is 4576 2024-03-29T10:55:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.773+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae8004eb0 con 0x7f4afc0d11e0 2024-03-29T10:55:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 client.?.objecter ms_handle_connect 0x7f4afc038c60 2024-03-29T10:55:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:41.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531462162 0 0) 0x7f4ae8004eb0 con 0x7f4afc0d11e0 2024-03-29T10:55:41.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4afc036b40 con 0x7f4afc0d11e0 2024-03-29T10:55:41.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4b0404fdd0 con 0x7f4afc0d11e0 2024-03-29T10:55:41.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:41.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:41.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 monclient(hunting): dump: 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:41.758 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:41.759 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:41.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:41.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 client.?.objecter ms_handle_connect 0x7f4afc0324b0 2024-03-29T10:55:41.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.775+0000 7f4aeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:41.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.776+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503721163 0 0) 0x7f4afc036860 con 0x7f4afc0324b0 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.777+0000 7f4aeffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.777+0000 7f4aeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.777+0000 7f4aeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.777+0000 7f4aeffff640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.777+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae8004eb0 con 0x7f4afc0324b0 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 72649083 0 0) 0x7f4afc036b40 con 0x7f4afc0d11e0 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 >> v1:172.21.15.64:6790/0 conn(0x7f4afc038c60 legacy=0x7f4afc0c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 >> v1:172.21.15.64:6789/0 conn(0x7f4afc0324b0 legacy=0x7f4afc0be280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 1 monclient: found mon.b 2024-03-29T10:55:41.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:41.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:41.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc037320 con 0x7f4afc0d11e0 2024-03-29T10:55:41.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 10 monclient: _finish_auth 0 2024-03-29T10:55:41.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:41.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.778+0000 7f4aeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:41.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.779+0000 7f4b08a2b640 5 monclient: authenticate success, global_id 4576 2024-03-29T10:55:41.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.779+0000 7f4b08a2b640 10 monclient: _renew_subs 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.779+0000 7f4b08a2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.779+0000 7f4b08a2b640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc160a00 con 0x7f4afc0d11e0 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.780+0000 7f4b08a2b640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.780+0000 7f4b08a2b640 10 monclient: _renew_subs 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.780+0000 7f4b08a2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.780+0000 7f4b08a2b640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc0beb70 con 0x7f4afc0d11e0 2024-03-29T10:55:41.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.780+0000 7f4b08a2b640 1 librados: init done 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b0404f400 con 0x7f4afc0d11e0 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4aeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4b04075f10 con 0x7f4afc0d11e0 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4aeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4aeffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4aeffff640 10 monclient: dump: 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:41.764 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:41.765 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:41.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.782+0000 7f4b017fa640 4 set_mon_vals no callback set 2024-03-29T10:55:41.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.783+0000 7f4aedffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:41.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.783+0000 7f4aedffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:41.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.783+0000 7f4aedffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.783+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4afc007b90 con 0x7f4afc0d11e0 2024-03-29T10:55:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.786+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f4b04067bf0 con 0x7f4afc0d11e0 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4b037fe640 10 monclient: get_auth_request con 0x7f4ae806db60 auth_method 0 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f4b040ff4b0 con 0x7f4afc0d11e0 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4aeffff640 10 client.4576.objecter ms_dispatch 0x7f4b04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4aeffff640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4aeffff640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4aeffff640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-29T10:55:41.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.787+0000 7f4aeffff640 10 client.4576.objecter ms_handle_connect 0x7f4ae806db60 2024-03-29T10:55:41.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.792+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b040cad90 con 0x7f4afc0d11e0 2024-03-29T10:55:41.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.792+0000 7f4aeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:41.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:41.792+0000 7f4aeffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:42.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.119+0000 7f4aedffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:42.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.119+0000 7f4aedffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:42.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.119+0000 7f4aedffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:55:42.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.119+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4afc037040 con 0x7f4afc0d11e0 2024-03-29T10:55:42.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.120+0000 7f4aeffff640 1 -- 172.21.15.64:0/2798000868 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f4b040cccc0 con 0x7f4afc0d11e0 2024-03-29T10:55:42.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.120+0000 7f4aeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:42.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.120+0000 7f4aeffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:42.103 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:42.103 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:42.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.124+0000 7f4aedffb640 10 librados: watch_flush enter 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 10 librados: watch_flush exit 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 10 client.4576.objecter successfully canceled tick 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 >> v1:172.21.15.174:6816/2473791532 conn(0x7f4ae806db60 legacy=0x7f4ae8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 10 monclient: shutdown 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.125+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 >> v1:172.21.15.174:6789/0 conn(0x7f4afc0d11e0 legacy=0x7f4afc0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.127+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 shutdown_connections 2024-03-29T10:55:42.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.127+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 >> 172.21.15.64:0/2798000868 conn(0x7f4afc005c20 msgr2=0x7f4afc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:42.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.127+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 shutdown_connections 2024-03-29T10:55:42.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.128+0000 7f4aedffb640 1 -- 172.21.15.64:0/2798000868 wait complete. 2024-03-29T10:55:42.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:42.128+0000 7f4aedffb640 1 librados: shutdown 2024-03-29T10:55:42.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:48.934 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:49.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.100+0000 7f01b96ef640 10 monclient: get_monmap_and_config 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.100+0000 7f01b96ef640 10 monclient: build_initial_monmap 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.100+0000 7f01b96ef640 10 monclient: monmap: 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:49.102725+0000 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:49.102725+0000 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:49.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.102+0000 7f01b96ef640 1 Processor -- start 2024-03-29T10:55:49.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.102+0000 7f01b96ef640 1 -- start start 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.102+0000 7f01b96ef640 10 monclient: init 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient(hunting): picked mon.noname-c con 0x7f01b40a9ac0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient(hunting): picked mon.noname-a con 0x7f01b413ee90 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient(hunting): picked mon.noname-b con 0x7f01b40af900 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40afd10 con 0x7f01b413ee90 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40af6f0 con 0x7f01b40a9ac0 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40aac80 con 0x7f01b40af900 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:49.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.103+0000 7f01b96ef640 10 monclient(hunting): authenticate will time out at 1360.403468s 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.108+0000 7f01b37fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f01b40af900 0x7f01b4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36535/0 (socket says 172.21.15.64:36535) 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.108+0000 7f01b37fe640 1 -- 172.21.15.64:0/862339838 learned_addr learned my addr 172.21.15.64:0/862339838 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.109+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157315826 0 0) 0x7f01b40aac80 con 0x7f01b40af900 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.109+0000 7f01b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.109+0000 7f01b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.109+0000 7f01b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.109+0000 7f01b1ffb640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:55:49.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.109+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019c003520 con 0x7f01b40af900 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404934995 0 0) 0x7f019c003520 con 0x7f01b40af900 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 >> v1:172.21.15.174:6789/0 conn(0x7f01b40a9ac0 legacy=0x7f01b40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 >> v1:172.21.15.64:6789/0 conn(0x7f01b413ee90 legacy=0x7f01b413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 monclient: found mon.noname-b 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b40aaf60 con 0x7f01b40af900 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:49.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b96ef640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b96ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f01a8001690 con 0x7f01b40af900 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: dump: 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:49.097 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b96ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 1 -- 172.21.15.64:0/862339838 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a80018b0 con 0x7f01b40af900 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b96ef640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.113+0000 7f01b96ef640 4 set_mon_vals no callback set 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 10 monclient: shutdown 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 1 -- 172.21.15.64:0/862339838 >> v1:172.21.15.64:6790/0 conn(0x7f01b40af900 legacy=0x7f01b4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 1 -- 172.21.15.64:0/862339838 shutdown_connections 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 1 -- 172.21.15.64:0/862339838 >> 172.21.15.64:0/862339838 conn(0x7f01b40b35e0 msgr2=0x7f01b40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:49.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 1 -- 172.21.15.64:0/862339838 shutdown_connections 2024-03-29T10:55:49.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.114+0000 7f01b96ef640 1 -- 172.21.15.64:0/862339838 wait complete. 2024-03-29T10:55:49.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.115+0000 7f01b96ef640 10 monclient: build_initial_monmap 2024-03-29T10:55:49.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.115+0000 7f01b96ef640 10 monclient: monmap: 2024-03-29T10:55:49.099 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:49.099 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:49.099 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 1 librados: starting msgr at 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 1 librados: starting objecter 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 1 Processor -- start 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 1 -- start start 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 1 librados: setting wanted keys 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 1 librados: calling monclient init 2024-03-29T10:55:49.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.116+0000 7f01b96ef640 10 monclient: init 2024-03-29T10:55:49.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:49.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:49.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient(hunting): picked mon.noname-b con 0x7f01b40a9ac0 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:49.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient(hunting): picked mon.noname-a con 0x7f01b40af900 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient(hunting): picked mon.noname-c con 0x7f01b413ee90 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b41e8170 con 0x7f01b40af900 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b41e7e50 con 0x7f01b40a9ac0 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b4071e40 con 0x7f01b413ee90 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.117+0000 7f01b96ef640 10 monclient(hunting): authenticate will time out at 1360.417339s 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f01b37fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f01b413ee90 0x7f01b41e1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36641/0 (socket says 172.21.15.64:36641) 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f01b37fe640 1 -- 172.21.15.64:0/3960744765 learned_addr learned my addr 172.21.15.64:0/3960744765 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 client.?.objecter ms_handle_connect 0x7f01b413ee90 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871591365 0 0) 0x7f01b4071e40 con 0x7f01b413ee90 2024-03-29T10:55:49.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:49.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:49.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:49.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:55:49.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a0004ee0 con 0x7f01b413ee90 2024-03-29T10:55:49.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566250112 0 0) 0x7f01a0004ee0 con 0x7f01b413ee90 2024-03-29T10:55:49.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01b4071e40 con 0x7f01b413ee90 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f01a8004ab0 con 0x7f01b413ee90 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:49.106 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.120+0000 7f019b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2854223689 0 0) 0x7f01b4071e40 con 0x7f01b413ee90 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 >> v1:172.21.15.174:6789/0 conn(0x7f01b40a9ac0 legacy=0x7f01b41e7740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 >> v1:172.21.15.64:6789/0 conn(0x7f01b40af900 legacy=0x7f01b41e5640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 1 monclient: found mon.c 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:49.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b41e8600 con 0x7f01b413ee90 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 10 monclient: _renew_subs 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b41e5d50 con 0x7f01b413ee90 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 10 monclient: _renew_subs 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b41e1aa0 con 0x7f01b413ee90 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b96ef640 1 librados: init done 2024-03-29T10:55:49.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a8001aa0 con 0x7f01b413ee90 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f019b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.121+0000 7f01b17fa640 4 set_mon_vals no callback set 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.122+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f01a8005060 con 0x7f01b413ee90 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.122+0000 7f019b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.122+0000 7f019b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.122+0000 7f019b7fe640 10 monclient: dump: 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:49.109 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.123+0000 7f01b96ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:49.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.123+0000 7f01b96ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.123+0000 7f01b96ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.123+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194005080 con 0x7f01b413ee90 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.126+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f01a801b030 con 0x7f01b413ee90 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.127+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f01a8084cd0 con 0x7f01b413ee90 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.127+0000 7f019b7fe640 10 client.4619.objecter ms_dispatch 0x7f01b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.127+0000 7f019b7fe640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.127+0000 7f019b7fe640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:49.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.127+0000 7f019b7fe640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T10:55:49.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.131+0000 7f01b2ffd640 10 monclient: get_auth_request con 0x7f01a006dc70 auth_method 0 2024-03-29T10:55:49.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.134+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01a80505b0 con 0x7f01b413ee90 2024-03-29T10:55:49.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.134+0000 7f019b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:49.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.134+0000 7f019b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:49.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.134+0000 7f019b7fe640 10 client.4619.objecter ms_handle_connect 0x7f01a006dc70 2024-03-29T10:55:49.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.471+0000 7f01b96ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:49.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.471+0000 7f01b96ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:49.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.471+0000 7f01b96ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:49.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.471+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0194005330 con 0x7f01b413ee90 2024-03-29T10:55:49.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.473+0000 7f019b7fe640 1 -- 172.21.15.64:0/3960744765 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f01a80524e0 con 0x7f01b413ee90 2024-03-29T10:55:49.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.473+0000 7f019b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:49.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.473+0000 7f019b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:49.455 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:49.455 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:49.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.476+0000 7f01b96ef640 10 librados: watch_flush enter 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 10 librados: watch_flush exit 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 10 client.4619.objecter successfully canceled tick 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 >> v1:172.21.15.174:6816/2473791532 conn(0x7f01a006dc70 legacy=0x7f01a0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 10 monclient: shutdown 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 >> v1:172.21.15.64:6790/0 conn(0x7f01b413ee90 legacy=0x7f01b41e1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 shutdown_connections 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.477+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 >> 172.21.15.64:0/3960744765 conn(0x7f01b40b35e0 msgr2=0x7f01b40ace90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:49.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.478+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 shutdown_connections 2024-03-29T10:55:49.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.478+0000 7f01b96ef640 1 -- 172.21.15.64:0/3960744765 wait complete. 2024-03-29T10:55:49.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:49.478+0000 7f01b96ef640 1 librados: shutdown 2024-03-29T10:55:49.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:56.285 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:56.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.454+0000 7f151c7f6640 10 monclient: get_monmap_and_config 2024-03-29T10:55:56.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.454+0000 7f151c7f6640 10 monclient: build_initial_monmap 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.454+0000 7f151c7f6640 10 monclient: monmap: 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:55:56.455729+0000 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:55:56.455729+0000 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:55:56.436 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.455+0000 7f151c7f6640 1 Processor -- start 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.455+0000 7f151c7f6640 1 -- start start 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.455+0000 7f151c7f6640 10 monclient: init 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151c7f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151c7f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151c7f6640 10 monclient(hunting): picked mon.noname-c con 0x7f15140b4710 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151c7f6640 10 monclient(hunting): picked mon.noname-b con 0x7f15140b3970 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151c7f6640 10 monclient(hunting): picked mon.noname-a con 0x7f151413f390 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151a592640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f15140b4710 0x7f15140b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:45694/0 (socket says 172.21.15.64:45694) 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151c7f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15140af020 con 0x7f151413f390 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.457+0000 7f151a592640 1 -- 172.21.15.64:0/2434279359 learned_addr learned my addr 172.21.15.64:0/2434279359 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:56.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15140b0910 con 0x7f15140b4710 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15140b0bf0 con 0x7f15140b3970 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f151c7f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f151c7f6640 10 monclient(hunting): authenticate will time out at 1367.756892s 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937588905 0 0) 0x7f15140b0bf0 con 0x7f15140b3970 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f1519590640 10 monclient(hunting): _negotiate 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f1519590640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f1519590640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.458+0000 7f1519590640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1500003520 con 0x7f15140b3970 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827241505 0 0) 0x7f15140b0910 con 0x7f15140b4710 2024-03-29T10:55:56.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): _negotiate 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): my global_id is 4582 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15140b0bf0 con 0x7f15140b4710 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164037381 0 0) 0x7f15140af020 con 0x7f151413f390 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): _negotiate 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15140b0910 con 0x7f151413f390 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 244508073 0 0) 0x7f15140b0bf0 con 0x7f15140b4710 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 >> v1:172.21.15.64:6790/0 conn(0x7f15140b3970 legacy=0x7f15140af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 >> v1:172.21.15.64:6789/0 conn(0x7f151413f390 legacy=0x7f151413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: _finish_hunting 0 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 monclient: found mon.noname-c 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15140aeb00 con 0x7f15140b4710 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: _finish_auth 0 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: _check_auth_tickets 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1504001670 con 0x7f15140b4710 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f1519590640 10 monclient: dump: 2024-03-29T10:55:56.444 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f151c7f6640 5 monclient: authenticate success, global_id 4582 2024-03-29T10:55:56.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.459+0000 7f151c7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15040017d0 con 0x7f15140b4710 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f1519590640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f1519590640 1 -- 172.21.15.64:0/2434279359 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1504001be0 con 0x7f15140b4710 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f1519590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f1519590640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f1519590640 10 monclient: dump: 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:56.457 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f151c7f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.461+0000 7f151c7f6640 4 set_mon_vals no callback set 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 10 monclient: shutdown 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 >> v1:172.21.15.174:6789/0 conn(0x7f15140b4710 legacy=0x7f15140b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 shutdown_connections 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 >> 172.21.15.64:0/2434279359 conn(0x7f1514066a00 msgr2=0x7f1514066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 shutdown_connections 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.462+0000 7f151c7f6640 1 -- 172.21.15.64:0/2434279359 wait complete. 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.463+0000 7f151c7f6640 10 monclient: build_initial_monmap 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.463+0000 7f151c7f6640 10 monclient: monmap: 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:56.459 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 1 librados: starting msgr at 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 1 librados: starting objecter 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 1 Processor -- start 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 1 -- start start 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 1 librados: setting wanted keys 2024-03-29T10:55:56.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 1 librados: calling monclient init 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.464+0000 7f151c7f6640 10 monclient: init 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient(hunting): picked mon.noname-c con 0x7f15140b3970 addr v1:172.21.15.64:6790/0 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient(hunting): picked mon.noname-a con 0x7f15140b4710 addr v1:172.21.15.64:6789/0 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient(hunting): picked mon.noname-b con 0x7f151413f390 addr v1:172.21.15.174:6789/0 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1514136870 con 0x7f15140b4710 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1514136b50 con 0x7f151413f390 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1514136e30 con 0x7f15140b3970 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151c7f6640 10 monclient(hunting): authenticate will time out at 1367.764556s 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151a592640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f15140b3970 0x7f1514141a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54050/0 (socket says 172.21.15.64:54050) 2024-03-29T10:55:56.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.465+0000 7f151a592640 1 -- 172.21.15.64:0/3123569614 learned_addr learned my addr 172.21.15.64:0/3123569614 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 client.?.objecter ms_handle_connect 0x7f15140b4710 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 client.?.objecter ms_handle_connect 0x7f15140b3970 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031942412 0 0) 0x7f1514136e30 con 0x7f15140b3970 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f8004eb0 con 0x7f15140b3970 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842164180 0 0) 0x7f1514136870 con 0x7f15140b4710 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:56.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.466+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1514136e30 con 0x7f15140b4710 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2489731299 0 0) 0x7f14f8004eb0 con 0x7f15140b3970 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1514136870 con 0x7f15140b3970 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1504001ba0 con 0x7f15140b3970 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 10 monclient(hunting): dump: 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.463 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.467+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273540050 0 0) 0x7f1514136e30 con 0x7f15140b4710 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14f8004eb0 con 0x7f15140b4710 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f15100018d0 con 0x7f15140b4710 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:56.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient(hunting): dump: 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1631015354 0 0) 0x7f1514136870 con 0x7f15140b3970 2024-03-29T10:55:56.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 >> v1:172.21.15.174:6789/0 conn(0x7f151413f390 legacy=0x7f15141e2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 >> v1:172.21.15.64:6789/0 conn(0x7f15140b4710 legacy=0x7f15141dedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 monclient: found mon.c 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514138620 con 0x7f15140b3970 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient: _finish_auth 0 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 10 monclient: _renew_subs 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:56.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1514134ab0 con 0x7f15140b3970 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 10 monclient: _renew_subs 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1514142120 con 0x7f15140b3970 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.468+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1504004a90 con 0x7f15140b3970 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f151c7f6640 1 librados: init done 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f150affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f1518d8f640 4 set_mon_vals no callback set 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1504004e20 con 0x7f15140b3970 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f150affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f150affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.469+0000 7f150affd640 10 monclient: dump: 2024-03-29T10:55:56.467 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f1504004f80 con 0x7f15140b3970 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f151c7f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f151c7f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:56.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f151c7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc005080 con 0x7f15140b3970 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f1519d91640 10 monclient: get_auth_request con 0x7f14f806dfd0 auth_method 0 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.471+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f1504084bc0 con 0x7f15140b3970 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.472+0000 7f150affd640 10 client.4625.objecter ms_dispatch 0x7f1514000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.472+0000 7f150affd640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.472+0000 7f150affd640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:56.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.472+0000 7f150affd640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T10:55:56.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.472+0000 7f150affd640 10 client.4625.objecter ms_handle_connect 0x7f14f806dfd0 2024-03-29T10:55:56.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.480+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15040504a0 con 0x7f15140b3970 2024-03-29T10:55:56.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.480+0000 7f150affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:56.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.480+0000 7f150affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:56.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.812+0000 7f151c7f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:56.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.812+0000 7f151c7f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:56.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.812+0000 7f151c7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:55:56.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.812+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14fc005330 con 0x7f15140b3970 2024-03-29T10:55:56.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.814+0000 7f150affd640 1 -- 172.21.15.64:0/3123569614 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f15040523d0 con 0x7f15140b3970 2024-03-29T10:55:56.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.814+0000 7f150affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:56.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.814+0000 7f150affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:56.796 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:55:56.796 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:56.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.818+0000 7f151c7f6640 10 librados: watch_flush enter 2024-03-29T10:55:56.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 10 librados: watch_flush exit 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 10 client.4625.objecter successfully canceled tick 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 >> v1:172.21.15.174:6816/2473791532 conn(0x7f14f806dfd0 legacy=0x7f14f8070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 10 monclient: shutdown 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 >> v1:172.21.15.64:6790/0 conn(0x7f15140b3970 legacy=0x7f1514141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 shutdown_connections 2024-03-29T10:55:56.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.819+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 >> 172.21.15.64:0/3123569614 conn(0x7f1514066a00 msgr2=0x7f15140b6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:56.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.820+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 shutdown_connections 2024-03-29T10:55:56.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.820+0000 7f151c7f6640 1 -- 172.21.15.64:0/3123569614 wait complete. 2024-03-29T10:55:56.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:55:56.820+0000 7f151c7f6640 1 librados: shutdown 2024-03-29T10:55:56.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:03.628 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:03.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.779+0000 7f505c82e640 10 monclient: get_monmap_and_config 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.779+0000 7f505c82e640 10 monclient: build_initial_monmap 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.779+0000 7f505c82e640 10 monclient: monmap: 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:03.780743+0000 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:03.780743+0000 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:03.761 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.780+0000 7f505c82e640 1 Processor -- start 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.780+0000 7f505c82e640 1 -- start start 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.780+0000 7f505c82e640 10 monclient: init 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient(hunting): picked mon.noname-b con 0x7f50580b4330 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:03.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient(hunting): picked mon.noname-c con 0x7f50581470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient(hunting): picked mon.noname-a con 0x7f50581439b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505806f2a0 con 0x7f50581439b0 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058146b90 con 0x7f50581470b0 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580b2ca0 con 0x7f50580b4330 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505c82e640 10 monclient(hunting): authenticate will time out at 1375.081018s 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505659c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f50580b4330 0x7f50580b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56936/0 (socket says 172.21.15.64:56936) 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505659c640 1 -- 172.21.15.64:0/1948445954 learned_addr learned my addr 172.21.15.64:0/1948445954 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685857841 0 0) 0x7f50580b2ca0 con 0x7f50580b4330 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:56:03.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044003520 con 0x7f50580b4330 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107854143 0 0) 0x7f5058146b90 con 0x7f50581470b0 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 10 monclient(hunting): my global_id is 4585 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.782+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50580b2ca0 con 0x7f50581470b0 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.783+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556490060 0 0) 0x7f5044003520 con 0x7f50580b4330 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.783+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 >> v1:172.21.15.174:6789/0 conn(0x7f50581470b0 legacy=0x7f50580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:03.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.783+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 >> v1:172.21.15.64:6789/0 conn(0x7f50581439b0 legacy=0x7f5058151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:03.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.784+0000 7f505559a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:03.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.784+0000 7f505559a640 1 monclient: found mon.noname-b 2024-03-29T10:56:03.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.784+0000 7f505559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:03.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.784+0000 7f505559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:56:03.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.784+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505806e0c0 con 0x7f50580b4330 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: _finish_auth 0 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5040001690 con 0x7f50580b4330 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: dump: 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:03.770 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505c82e640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505c82e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50400018b0 con 0x7f50580b4330 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 1 -- 172.21.15.64:0/1948445954 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5040001c40 con 0x7f50580b4330 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.785+0000 7f505559a640 10 monclient: dump: 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:03.771 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.786+0000 7f505c82e640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.786+0000 7f505c82e640 4 set_mon_vals no callback set 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 10 monclient: shutdown 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 1 -- 172.21.15.64:0/1948445954 >> v1:172.21.15.64:6790/0 conn(0x7f50580b4330 legacy=0x7f50580b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 1 -- 172.21.15.64:0/1948445954 shutdown_connections 2024-03-29T10:56:03.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 1 -- 172.21.15.64:0/1948445954 >> 172.21.15.64:0/1948445954 conn(0x7f5058066a00 msgr2=0x7f5058066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:03.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 1 -- 172.21.15.64:0/1948445954 shutdown_connections 2024-03-29T10:56:03.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.787+0000 7f505c82e640 1 -- 172.21.15.64:0/1948445954 wait complete. 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.788+0000 7f505c82e640 10 monclient: build_initial_monmap 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.788+0000 7f505c82e640 10 monclient: monmap: 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:03.779 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 1 librados: starting msgr at 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 1 librados: starting objecter 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 1 Processor -- start 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 1 -- start start 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 1 librados: setting wanted keys 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 1 librados: calling monclient init 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.789+0000 7f505c82e640 10 monclient: init 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.790+0000 7f505c82e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.790+0000 7f505c82e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:03.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.790+0000 7f505c82e640 10 monclient(hunting): picked mon.noname-b con 0x7f50580b4330 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 10 monclient(hunting): picked mon.noname-c con 0x7f50581439b0 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f5055d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f50581439b0 0x7f5058125160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:57296/0 (socket says 172.21.15.64:57296) 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f5055d9b640 1 -- 172.21.15.64:0/1208048976 learned_addr learned my addr 172.21.15.64:0/1208048976 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 client.?.objecter ms_handle_connect 0x7f50581439b0 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 client.?.objecter ms_handle_connect 0x7f50580b4330 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 10 monclient(hunting): picked mon.noname-a con 0x7f50581470b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058127600 con 0x7f50581470b0 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058125870 con 0x7f50580b4330 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50581224c0 con 0x7f50581439b0 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f505c82e640 10 monclient(hunting): authenticate will time out at 1375.090472s 2024-03-29T10:56:03.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876656522 0 0) 0x7f50581224c0 con 0x7f50581439b0 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5038004eb0 con 0x7f50581439b0 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803613216 0 0) 0x7f5058125870 con 0x7f50580b4330 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 10 monclient(hunting): my global_id is 4588 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.791+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50581224c0 con 0x7f50580b4330 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 207699200 0 0) 0x7f5038004eb0 con 0x7f50581439b0 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5058125870 con 0x7f50581439b0 2024-03-29T10:56:03.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5048001d90 con 0x7f50581439b0 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 10 monclient(hunting): dump: 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:03.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1588287862 0 0) 0x7f5058125870 con 0x7f50581439b0 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 >> v1:172.21.15.174:6789/0 conn(0x7f50580b4330 legacy=0x7f5058125f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.792+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 >> v1:172.21.15.64:6789/0 conn(0x7f50581470b0 legacy=0x7f50581deff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 1 monclient: found mon.c 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5058122650 con 0x7f50581439b0 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 10 monclient: _finish_auth 0 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f504effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 10 monclient: _renew_subs 2024-03-29T10:56:03.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50581df700 con 0x7f50581439b0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 10 monclient: _renew_subs 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581e7230 con 0x7f50581439b0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.793+0000 7f505c82e640 1 librados: init done 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f505c82e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f505c82e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f505c82e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5058075f10 con 0x7f50581439b0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5048001fd0 con 0x7f50581439b0 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f504effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:03.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f5048003de0 con 0x7f50581439b0 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f504effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f504effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f504effd640 10 monclient: dump: 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:03.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.795+0000 7f5054d99640 4 set_mon_vals no callback set 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.796+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f5048004000 con 0x7f50581439b0 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.797+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1775078084 0 0) 0x7f5048084e80 con 0x7f50581439b0 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.797+0000 7f504effd640 10 client.4631.objecter ms_dispatch 0x7f5058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.797+0000 7f504effd640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.797+0000 7f504effd640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.798+0000 7f505659c640 10 monclient: get_auth_request con 0x7f503806dd70 auth_method 0 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.798+0000 7f504effd640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T10:56:03.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.798+0000 7f504effd640 10 client.4631.objecter ms_handle_connect 0x7f503806dd70 2024-03-29T10:56:03.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.804+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5048050760 con 0x7f50581439b0 2024-03-29T10:56:03.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.804+0000 7f504effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:03.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:03.804+0000 7f504effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:04.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.111+0000 7f505c82e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:04.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.111+0000 7f505c82e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:04.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.111+0000 7f505c82e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:04.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.111+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50581ded10 con 0x7f50581439b0 2024-03-29T10:56:04.095 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:04.095 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:46:06.668045+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:06.668045+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:04.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.112+0000 7f504effd640 1 -- 172.21.15.64:0/1208048976 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1986663607) 0x7f50581ded10 con 0x7f50581439b0 2024-03-29T10:56:04.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.112+0000 7f504effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:04.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.112+0000 7f504effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:04.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.114+0000 7f505c82e640 10 librados: watch_flush enter 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.114+0000 7f505c82e640 10 librados: watch_flush exit 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.114+0000 7f505c82e640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.114+0000 7f505c82e640 10 client.4631.objecter successfully canceled tick 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 >> v1:172.21.15.174:6816/2473791532 conn(0x7f503806dd70 legacy=0x7f5038070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 10 monclient: shutdown 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 >> v1:172.21.15.64:6790/0 conn(0x7f50581439b0 legacy=0x7f5058125160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 shutdown_connections 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 >> 172.21.15.64:0/1208048976 conn(0x7f5058066a00 msgr2=0x7f50580b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 shutdown_connections 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 -- 172.21.15.64:0/1208048976 wait complete. 2024-03-29T10:56:04.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:04.115+0000 7f505c82e640 1 librados: shutdown 2024-03-29T10:56:04.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:10.916 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.087+0000 7fcc6da88640 10 monclient: get_monmap_and_config 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.087+0000 7fcc6da88640 10 monclient: build_initial_monmap 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.087+0000 7fcc6da88640 10 monclient: monmap: 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:11.087889+0000 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:11.087889+0000 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:11.068 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.088+0000 7fcc6da88640 1 Processor -- start 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.088+0000 7fcc6da88640 1 -- start start 2024-03-29T10:56:11.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.088+0000 7fcc6da88640 10 monclient: init 2024-03-29T10:56:11.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.089+0000 7fcc6da88640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:11.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.089+0000 7fcc6da88640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:11.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.090+0000 7fcc6da88640 10 monclient(hunting): picked mon.noname-b con 0x7fcc680b0770 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:11.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 10 monclient(hunting): picked mon.noname-a con 0x7fcc6814a700 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:11.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 10 monclient(hunting): picked mon.noname-c con 0x7fcc680ac5a0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:11.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680aba70 con 0x7fcc6814a700 2024-03-29T10:56:11.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680aecb0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680af150 con 0x7fcc680b0770 2024-03-29T10:56:11.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:11.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.091+0000 7fcc6da88640 10 monclient(hunting): authenticate will time out at 1382.389957s 2024-03-29T10:56:11.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.092+0000 7fcc677fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcc680b0770 0x7fcc68149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42350/0 (socket says 172.21.15.64:42350) 2024-03-29T10:56:11.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.092+0000 7fcc677fe640 1 -- 172.21.15.64:0/271454344 learned_addr learned my addr 172.21.15.64:0/271454344 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:11.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473716085 0 0) 0x7fcc680af150 con 0x7fcc680b0770 2024-03-29T10:56:11.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:11.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:11.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:11.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:56:11.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc50003520 con 0x7fcc680b0770 2024-03-29T10:56:11.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048458560 0 0) 0x7fcc50003520 con 0x7fcc680b0770 2024-03-29T10:56:11.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.094+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 >> v1:172.21.15.174:6789/0 conn(0x7fcc680ac5a0 legacy=0x7fcc68151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:11.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 >> v1:172.21.15.64:6789/0 conn(0x7fcc6814a700 legacy=0x7fcc680abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:11.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 1 monclient: found mon.noname-b 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680ac9d0 con 0x7fcc680b0770 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcc58001690 con 0x7fcc680b0770 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.097+0000 7fcc6da88640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc6da88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:11.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 10 monclient: dump: 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc6da88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:11.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc58001860 con 0x7fcc680b0770 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 1 -- 172.21.15.64:0/271454344 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcc58001c70 con 0x7fcc680b0770 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.098+0000 7fcc667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.099+0000 7fcc667fc640 10 monclient: dump: 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:11.084 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.099+0000 7fcc6da88640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.099+0000 7fcc6da88640 4 set_mon_vals no callback set 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.100+0000 7fcc6da88640 10 monclient: shutdown 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.100+0000 7fcc6da88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.100+0000 7fcc6da88640 1 -- 172.21.15.64:0/271454344 >> v1:172.21.15.64:6790/0 conn(0x7fcc680b0770 legacy=0x7fcc68149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.101+0000 7fcc6da88640 1 -- 172.21.15.64:0/271454344 shutdown_connections 2024-03-29T10:56:11.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.101+0000 7fcc6da88640 1 -- 172.21.15.64:0/271454344 >> 172.21.15.64:0/271454344 conn(0x7fcc6806b660 msgr2=0x7fcc6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:11.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.101+0000 7fcc6da88640 1 -- 172.21.15.64:0/271454344 shutdown_connections 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.101+0000 7fcc6da88640 1 -- 172.21.15.64:0/271454344 wait complete. 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.103+0000 7fcc6da88640 10 monclient: build_initial_monmap 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.103+0000 7fcc6da88640 10 monclient: monmap: 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:11.089 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.103+0000 7fcc6da88640 1 librados: starting msgr at 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.103+0000 7fcc6da88640 1 librados: starting objecter 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.103+0000 7fcc6da88640 1 Processor -- start 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.104+0000 7fcc6da88640 1 -- start start 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.104+0000 7fcc6da88640 1 librados: setting wanted keys 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.104+0000 7fcc6da88640 1 librados: calling monclient init 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.104+0000 7fcc6da88640 10 monclient: init 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.105+0000 7fcc6da88640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.105+0000 7fcc6da88640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.105+0000 7fcc6da88640 10 monclient(hunting): picked mon.noname-c con 0x7fcc680ac5a0 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:11.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.105+0000 7fcc6da88640 10 monclient(hunting): picked mon.noname-a con 0x7fcc680b0770 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc6da88640 10 monclient(hunting): picked mon.noname-b con 0x7fcc6814a700 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc6da88640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68128550 con 0x7fcc680b0770 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc6da88640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68126740 con 0x7fcc6814a700 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc6da88640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc681239d0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc6da88640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc6da88640 10 monclient(hunting): authenticate will time out at 1382.406107s 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.105+0000 7fcc677fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcc680ac5a0 0x7fcc681450d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:43466/0 (socket says 172.21.15.64:43466) 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc677fe640 1 -- 172.21.15.64:0/3180180356 learned_addr learned my addr 172.21.15.64:0/3180180356 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 10 client.?.objecter ms_handle_connect 0x7fcc680ac5a0 2024-03-29T10:56:11.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:11.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356651669 0 0) 0x7fcc681239d0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:11.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:11.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:11.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.107+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc54004eb0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1762627419 0 0) 0x7fcc54004eb0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc681239d0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcc58001b70 con 0x7fcc680ac5a0 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 10 monclient(hunting): dump: 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:11.094 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1463586306 0 0) 0x7fcc681239d0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 >> v1:172.21.15.174:6789/0 conn(0x7fcc6814a700 legacy=0x7fcc681e7100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.108+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 >> v1:172.21.15.64:6789/0 conn(0x7fcc680b0770 legacy=0x7fcc68126c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 1 monclient: found mon.c 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc681260f0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 10 monclient: _finish_auth 0 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 5 monclient: authenticate success, global_id 4637 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 10 monclient: _renew_subs 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc681263a0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 10 monclient: _renew_subs 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc681dfbc0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.109+0000 7fcc6da88640 1 librados: init done 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc58004bf0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:11.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fcc58005040 con 0x7fcc680ac5a0 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc4ffff640 10 monclient: dump: 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc65ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:11.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.110+0000 7fcc6da88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:11.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.111+0000 7fcc6da88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:11.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.111+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fcc580051a0 con 0x7fcc680ac5a0 2024-03-29T10:56:11.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.112+0000 7fcc6da88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:11.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.112+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc68075f10 con 0x7fcc680ac5a0 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.113+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7fcc58084d40 con 0x7fcc680ac5a0 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.113+0000 7fcc4ffff640 10 client.4637.objecter ms_dispatch 0x7fcc68000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.113+0000 7fcc4ffff640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.113+0000 7fcc4ffff640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.113+0000 7fcc4ffff640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.113+0000 7fcc66ffd640 10 monclient: get_auth_request con 0x7fcc5406dd40 auth_method 0 2024-03-29T10:56:11.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.114+0000 7fcc4ffff640 10 client.4637.objecter ms_handle_connect 0x7fcc5406dd40 2024-03-29T10:56:11.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.120+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc58050620 con 0x7fcc680ac5a0 2024-03-29T10:56:11.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.120+0000 7fcc4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:11.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.120+0000 7fcc4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.428+0000 7fcc6da88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.428+0000 7fcc6da88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.428+0000 7fcc6da88640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.428+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc68126980 con 0x7fcc680ac5a0 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.429+0000 7fcc4ffff640 1 -- 172.21.15.64:0/3180180356 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7fcc58052550 con 0x7fcc680ac5a0 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.429+0000 7fcc4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:11.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.429+0000 7fcc4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:11.410 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:11.410 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:11.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 10 librados: watch_flush enter 2024-03-29T10:56:11.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 10 librados: watch_flush exit 2024-03-29T10:56:11.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 10 client.4637.objecter successfully canceled tick 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 >> v1:172.21.15.174:6816/2473791532 conn(0x7fcc5406dd40 legacy=0x7fcc54070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 10 monclient: shutdown 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 >> v1:172.21.15.64:6790/0 conn(0x7fcc680ac5a0 legacy=0x7fcc681450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 shutdown_connections 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.432+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 >> 172.21.15.64:0/3180180356 conn(0x7fcc6806b660 msgr2=0x7fcc680b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.433+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 shutdown_connections 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.433+0000 7fcc6da88640 1 -- 172.21.15.64:0/3180180356 wait complete. 2024-03-29T10:56:11.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:11.433+0000 7fcc6da88640 1 librados: shutdown 2024-03-29T10:56:11.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:18.235 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:18.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.404+0000 7f24e8f82640 10 monclient: get_monmap_and_config 2024-03-29T10:56:18.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.404+0000 7f24e8f82640 10 monclient: build_initial_monmap 2024-03-29T10:56:18.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.404+0000 7f24e8f82640 10 monclient: monmap: 2024-03-29T10:56:18.386 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:18.386 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:18.386 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:18.405537+0000 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:18.405537+0000 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.406+0000 7f24e8f82640 1 Processor -- start 2024-03-29T10:56:18.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.406+0000 7f24e8f82640 1 -- start start 2024-03-29T10:56:18.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.406+0000 7f24e8f82640 10 monclient: init 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient(hunting): picked mon.noname-b con 0x7f24e40b4100 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient(hunting): picked mon.noname-a con 0x7f24e413ee90 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient(hunting): picked mon.noname-c con 0x7f24e40a9ac0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e40af6f0 con 0x7f24e413ee90 2024-03-29T10:56:18.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e40aac80 con 0x7f24e40a9ac0 2024-03-29T10:56:18.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e40aaf60 con 0x7f24e40b4100 2024-03-29T10:56:18.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:18.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.408+0000 7f24e8f82640 10 monclient(hunting): authenticate will time out at 1389.707327s 2024-03-29T10:56:18.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.410+0000 7f24e259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f24e40b4100 0x7f24e40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47164/0 (socket says 172.21.15.64:47164) 2024-03-29T10:56:18.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.410+0000 7f24e259c640 1 -- 172.21.15.64:0/2768788759 learned_addr learned my addr 172.21.15.64:0/2768788759 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:18.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042242428 0 0) 0x7f24e40aaf60 con 0x7f24e40b4100 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24cc003520 con 0x7f24e40b4100 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856900504 0 0) 0x7f24e40af6f0 con 0x7f24e413ee90 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:56:18.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.412+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e40aaf60 con 0x7f24e413ee90 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3850565088 0 0) 0x7f24e40aaf60 con 0x7f24e413ee90 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 >> v1:172.21.15.64:6790/0 conn(0x7f24e40b4100 legacy=0x7f24e40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 >> v1:172.21.15.174:6789/0 conn(0x7f24e40a9ac0 legacy=0x7f24e4142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 monclient: found mon.noname-a 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:18.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e413e970 con 0x7f24e413ee90 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: _finish_auth 0 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e8f82640 5 monclient: authenticate success, global_id 4566 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e8f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f24dc0019f0 con 0x7f24e413ee90 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: dump: 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:18.398 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e8f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc001c10 con 0x7f24e413ee90 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e8f82640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.413+0000 7f24e8f82640 4 set_mon_vals no callback set 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e159a640 1 -- 172.21.15.64:0/2768788759 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f24dc001fa0 con 0x7f24e413ee90 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:18.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e159a640 10 monclient: dump: 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e8f82640 10 monclient: shutdown 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e8f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:18.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e8f82640 1 -- 172.21.15.64:0/2768788759 >> v1:172.21.15.64:6789/0 conn(0x7f24e413ee90 legacy=0x7f24e413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e8f82640 1 -- 172.21.15.64:0/2768788759 shutdown_connections 2024-03-29T10:56:18.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.414+0000 7f24e8f82640 1 -- 172.21.15.64:0/2768788759 >> 172.21.15.64:0/2768788759 conn(0x7f24e40b35e0 msgr2=0x7f24e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:18.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.415+0000 7f24e8f82640 1 -- 172.21.15.64:0/2768788759 shutdown_connections 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.416+0000 7f24e8f82640 1 -- 172.21.15.64:0/2768788759 wait complete. 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.417+0000 7f24e8f82640 10 monclient: build_initial_monmap 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.417+0000 7f24e8f82640 10 monclient: monmap: 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:18.404 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 1 librados: starting msgr at 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 1 librados: starting objecter 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 1 Processor -- start 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 1 -- start start 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 1 librados: setting wanted keys 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 1 librados: calling monclient init 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.418+0000 7f24e8f82640 10 monclient: init 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient(hunting): picked mon.noname-b con 0x7f24e40a9ac0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient(hunting): picked mon.noname-c con 0x7f24e40b4100 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:18.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient(hunting): picked mon.noname-a con 0x7f24e413ee90 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e41c92f0 con 0x7f24e413ee90 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e41c52d0 con 0x7f24e40a9ac0 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e41c8850 con 0x7f24e40b4100 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.419+0000 7f24e8f82640 10 monclient(hunting): authenticate will time out at 1389.718633s 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24e1d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f24e40b4100 0x7f24e4071100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50866/0 (socket says 172.21.15.64:50866) 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24e1d9b640 1 -- 172.21.15.64:0/255099212 learned_addr learned my addr 172.21.15.64:0/255099212 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 10 client.?.objecter ms_handle_connect 0x7f24e40b4100 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754561905 0 0) 0x7f24e41c8850 con 0x7f24e40b4100 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:18.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:18.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.420+0000 7f24d2ffd640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:56:18.407 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.421+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c4004eb0 con 0x7f24e40b4100 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.424+0000 7f24d2ffd640 10 client.?.objecter ms_handle_connect 0x7f24e40a9ac0 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.424+0000 7f24d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.424+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3105679743 0 0) 0x7f24c4004eb0 con 0x7f24e40b4100 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.424+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24e41c8850 con 0x7f24e40b4100 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.424+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081766723 0 0) 0x7f24e41c52d0 con 0x7f24e40a9ac0 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): my global_id is 4597 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c4004eb0 con 0x7f24e40a9ac0 2024-03-29T10:56:18.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f24dc001fe0 con 0x7f24e40b4100 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.425+0000 7f24d2ffd640 10 monclient(hunting): dump: 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:18.410 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 client.?.objecter ms_handle_connect 0x7f24e413ee90 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1451252590 0 0) 0x7f24e41c8850 con 0x7f24e40b4100 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 >> v1:172.21.15.174:6789/0 conn(0x7f24e40a9ac0 legacy=0x7f24e41d2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 >> v1:172.21.15.64:6789/0 conn(0x7f24e413ee90 legacy=0x7f24e41dd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 1 monclient: found mon.c 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e41c89e0 con 0x7f24e40b4100 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:18.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:18.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:56:18.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 10 monclient: _renew_subs 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e4071810 con 0x7f24e40b4100 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 10 monclient: _renew_subs 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e4071cf0 con 0x7f24e40b4100 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.426+0000 7f24e8f82640 1 librados: init done 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc003c20 con 0x7f24e40b4100 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f24dc003fb0 con 0x7f24e40b4100 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24d2ffd640 10 monclient: dump: 2024-03-29T10:56:18.413 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24e0d99640 4 set_mon_vals no callback set 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.428+0000 7f24e8f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.429+0000 7f24e8f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:18.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.429+0000 7f24e8f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.429+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e40aaec0 con 0x7f24e40b4100 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.431+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f24dc004110 con 0x7f24e40b4100 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.432+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7f24dc084e30 con 0x7f24e40b4100 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.433+0000 7f24e259c640 10 monclient: get_auth_request con 0x7f24c406de20 auth_method 0 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.433+0000 7f24d2ffd640 10 client.4643.objecter ms_dispatch 0x7f24e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.433+0000 7f24d2ffd640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.433+0000 7f24d2ffd640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:18.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.433+0000 7f24d2ffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T10:56:18.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.433+0000 7f24d2ffd640 10 client.4643.objecter ms_handle_connect 0x7f24c406de20 2024-03-29T10:56:18.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.439+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24dc050710 con 0x7f24e40b4100 2024-03-29T10:56:18.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.439+0000 7f24d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:18.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.439+0000 7f24d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:18.757 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.776+0000 7f24e8f82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:18.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.776+0000 7f24e8f82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:18.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.776+0000 7f24e8f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:18.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.776+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24e41c8b70 con 0x7f24e40b4100 2024-03-29T10:56:18.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.780+0000 7f24d2ffd640 1 -- 172.21.15.64:0/255099212 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7f24dc052640 con 0x7f24e40b4100 2024-03-29T10:56:18.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.780+0000 7f24d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:18.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.780+0000 7f24d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:18.761 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:18.762 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 10 librados: watch_flush enter 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 10 librados: watch_flush exit 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 10 client.4643.objecter successfully canceled tick 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 >> v1:172.21.15.174:6816/2473791532 conn(0x7f24c406de20 legacy=0x7f24c40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 10 monclient: shutdown 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 >> v1:172.21.15.64:6790/0 conn(0x7f24e40b4100 legacy=0x7f24e4071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 shutdown_connections 2024-03-29T10:56:18.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.784+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 >> 172.21.15.64:0/255099212 conn(0x7f24e40b35e0 msgr2=0x7f24e40b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:18.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.785+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 shutdown_connections 2024-03-29T10:56:18.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.785+0000 7f24e8f82640 1 -- 172.21.15.64:0/255099212 wait complete. 2024-03-29T10:56:18.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:18.785+0000 7f24e8f82640 1 librados: shutdown 2024-03-29T10:56:18.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:25.588 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:25.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.756+0000 7f1ab27bc640 10 monclient: get_monmap_and_config 2024-03-29T10:56:25.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.756+0000 7f1ab27bc640 10 monclient: build_initial_monmap 2024-03-29T10:56:25.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.756+0000 7f1ab27bc640 10 monclient: monmap: 2024-03-29T10:56:25.737 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:25.757685+0000 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:25.757685+0000 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:25.738 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:25.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.757+0000 7f1ab27bc640 1 Processor -- start 2024-03-29T10:56:25.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.758+0000 7f1ab27bc640 1 -- start start 2024-03-29T10:56:25.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.758+0000 7f1ab27bc640 10 monclient: init 2024-03-29T10:56:25.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.759+0000 7f1ab27bc640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:25.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.759+0000 7f1ab27bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:25.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.759+0000 7f1ab27bc640 10 monclient(hunting): picked mon.noname-c con 0x7f1aa40389e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:25.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.759+0000 7f1ab27bc640 10 monclient(hunting): picked mon.noname-b con 0x7f1aa4038dc0 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:25.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.762+0000 7f1ab27bc640 10 monclient(hunting): picked mon.noname-a con 0x7f1aa4030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:25.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.762+0000 7f1ab27bc640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa40c54c0 con 0x7f1aa4030660 2024-03-29T10:56:25.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.762+0000 7f1ab27bc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa4036430 con 0x7f1aa40389e0 2024-03-29T10:56:25.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.762+0000 7f1ab27bc640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa4039840 con 0x7f1aa4038dc0 2024-03-29T10:56:25.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.762+0000 7f1ab27bc640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:25.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.762+0000 7f1ab27bc640 10 monclient(hunting): authenticate will time out at 1397.061611s 2024-03-29T10:56:25.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1ab1fbb640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1aa4030660 0x7f1aa40d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:55197/0 (socket says 172.21.15.64:55197) 2024-03-29T10:56:25.744 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1ab1fbb640 1 -- 172.21.15.64:0/278102197 learned_addr learned my addr 172.21.15.64:0/278102197 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13422960 0 0) 0x7f1aa40c54c0 con 0x7f1aa4030660 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1aa3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1aa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1aa3fff640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.763+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98003520 con 0x7f1aa4030660 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.764+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1204506058 0 0) 0x7f1a98003520 con 0x7f1aa4030660 2024-03-29T10:56:25.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.764+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 >> v1:172.21.15.64:6790/0 conn(0x7f1aa4038dc0 legacy=0x7f1aa40cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:25.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.766+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 >> v1:172.21.15.174:6789/0 conn(0x7f1aa40389e0 legacy=0x7f1aa4033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 1 monclient: found mon.noname-a 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4035690 con 0x7f1aa4030660 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: _finish_auth 0 2024-03-29T10:56:25.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1aac04f730 con 0x7f1aa4030660 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1aa3fff640 10 monclient: dump: 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:25.749 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:25.750 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:25.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1ab27bc640 5 monclient: authenticate success, global_id 4572 2024-03-29T10:56:25.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.767+0000 7f1ab27bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1aac0008d0 con 0x7f1aa4030660 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1aa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1aa3fff640 1 -- 172.21.15.64:0/278102197 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1aac0758b0 con 0x7f1aa4030660 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1aa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1aa3fff640 10 monclient: dump: 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.751 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.752 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:25.752 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:25.752 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:25.752 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:25.752 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:25.752 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:25.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1ab27bc640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:25.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.770+0000 7f1ab27bc640 4 set_mon_vals no callback set 2024-03-29T10:56:25.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.771+0000 7f1ab27bc640 10 monclient: shutdown 2024-03-29T10:56:25.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.771+0000 7f1ab27bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:25.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.771+0000 7f1ab27bc640 1 -- 172.21.15.64:0/278102197 >> v1:172.21.15.64:6789/0 conn(0x7f1aa4030660 legacy=0x7f1aa40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:25.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.772+0000 7f1ab27bc640 1 -- 172.21.15.64:0/278102197 shutdown_connections 2024-03-29T10:56:25.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.772+0000 7f1ab27bc640 1 -- 172.21.15.64:0/278102197 >> 172.21.15.64:0/278102197 conn(0x7f1aa4005c00 msgr2=0x7f1aa4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:25.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.775+0000 7f1ab27bc640 1 -- 172.21.15.64:0/278102197 shutdown_connections 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.775+0000 7f1ab27bc640 1 -- 172.21.15.64:0/278102197 wait complete. 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.776+0000 7f1ab27bc640 10 monclient: build_initial_monmap 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.776+0000 7f1ab27bc640 10 monclient: monmap: 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:25.758 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.776+0000 7f1ab27bc640 1 librados: starting msgr at 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.776+0000 7f1ab27bc640 1 librados: starting objecter 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.777+0000 7f1ab27bc640 1 Processor -- start 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.777+0000 7f1ab27bc640 1 -- start start 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.777+0000 7f1ab27bc640 1 librados: setting wanted keys 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.777+0000 7f1ab27bc640 1 librados: calling monclient init 2024-03-29T10:56:25.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.777+0000 7f1ab27bc640 10 monclient: init 2024-03-29T10:56:25.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.778+0000 7f1ab27bc640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:25.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.778+0000 7f1ab27bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:25.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.778+0000 7f1ab27bc640 10 monclient(hunting): picked mon.noname-a con 0x7f1aa4030660 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab0fb9640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1aa40389e0 0x7f1aa416c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50669/0 (socket says 172.21.15.64:50669) 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 10 monclient(hunting): picked mon.noname-c con 0x7f1aa40389e0 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 10 monclient(hunting): picked mon.noname-b con 0x7f1aa415f970 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa40c6010 con 0x7f1aa4030660 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa40c5650 con 0x7f1aa415f970 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa416d070 con 0x7f1aa40389e0 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:25.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab27bc640 10 monclient(hunting): authenticate will time out at 1397.081002s 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1ab0fb9640 1 -- 172.21.15.64:0/2887211064 learned_addr learned my addr 172.21.15.64:0/2887211064 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f1aa40389e0 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408237306 0 0) 0x7f1aa416d070 con 0x7f1aa40389e0 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.782+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a90004eb0 con 0x7f1aa40389e0 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f1aa4030660 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844981695 0 0) 0x7f1aa40c6010 con 0x7f1aa4030660 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:25.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 10 monclient(hunting): my global_id is 4575 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.783+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa416d070 con 0x7f1aa4030660 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2819751214 0 0) 0x7f1aa416d070 con 0x7f1aa4030660 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a90005490 con 0x7f1aa4030660 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1aa8001810 con 0x7f1aa4030660 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 10 monclient(hunting): dump: 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.769 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.784+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2147387792 0 0) 0x7f1a90005490 con 0x7f1aa4030660 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 >> v1:172.21.15.64:6790/0 conn(0x7f1aa40389e0 legacy=0x7f1aa416c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 >> v1:172.21.15.174:6789/0 conn(0x7f1aa415f970 legacy=0x7f1aa40c6530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 1 monclient: found mon.a 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:25.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa416d200 con 0x7f1aa4030660 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.785+0000 7f1aa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.786+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1aa8001d00 con 0x7f1aa4030660 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.786+0000 7f1aa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.786+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f1aa8001810 con 0x7f1aa4030660 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.786+0000 7f1aa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.786+0000 7f1aa1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.786+0000 7f1aa1ffb640 10 monclient: dump: 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:25.771 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:25.772 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:25.772 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:25.772 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:25.772 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:25.772 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:25.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 5 monclient: authenticate success, global_id 4575 2024-03-29T10:56:25.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1aa37fe640 4 set_mon_vals no callback set 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 10 monclient: _renew_subs 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa416d3f0 con 0x7f1aa4030660 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 10 monclient: _renew_subs 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa40c6e20 con 0x7f1aa4030660 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.788+0000 7f1ab27bc640 1 librados: init done 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.791+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f1aa8019500 con 0x7f1aa4030660 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1ab0fb9640 10 monclient: get_auth_request con 0x7f1a9006e220 auth_method 0 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7f1aa8001630 con 0x7f1aa4030660 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1aa1ffb640 10 client.4575.objecter ms_dispatch 0x7f1aac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1aa1ffb640 3 client.4575.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:25.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1aa1ffb640 3 client.4575.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:25.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1aa1ffb640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T10:56:25.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1aa1ffb640 10 client.4575.objecter ms_handle_connect 0x7f1a9006e220 2024-03-29T10:56:25.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1ab27bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:25.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1ab27bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:25.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1ab27bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:25.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.792+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4007b90 con 0x7f1aa4030660 2024-03-29T10:56:25.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.801+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1aa804f3f0 con 0x7f1aa4030660 2024-03-29T10:56:25.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.801+0000 7f1aa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:25.782 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:25.801+0000 7f1aa1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:26.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.126+0000 7f1ab27bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:26.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.126+0000 7f1ab27bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:26.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.126+0000 7f1ab27bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:26.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.126+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aa4005120 con 0x7f1aa4030660 2024-03-29T10:56:26.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.129+0000 7f1aa1ffb640 1 -- 172.21.15.64:0/2887211064 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7f1aa804f590 con 0x7f1aa4030660 2024-03-29T10:56:26.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.129+0000 7f1aa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:26.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.129+0000 7f1aa1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:26.110 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:26.110 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:26.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.134+0000 7f1ab27bc640 10 librados: watch_flush enter 2024-03-29T10:56:26.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.135+0000 7f1ab27bc640 10 librados: watch_flush exit 2024-03-29T10:56:26.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.135+0000 7f1ab27bc640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T10:56:26.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.135+0000 7f1ab27bc640 10 client.4575.objecter successfully canceled tick 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 >> v1:172.21.15.174:6816/2473791532 conn(0x7f1a9006e220 legacy=0x7f1a900706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 10 monclient: shutdown 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 >> v1:172.21.15.64:6789/0 conn(0x7f1aa4030660 legacy=0x7f1aa415f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 shutdown_connections 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 >> 172.21.15.64:0/2887211064 conn(0x7f1aa4005c00 msgr2=0x7f1aa403ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 shutdown_connections 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 -- 172.21.15.64:0/2887211064 wait complete. 2024-03-29T10:56:26.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:26.136+0000 7f1ab27bc640 1 librados: shutdown 2024-03-29T10:56:26.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:32.941 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:33.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.102+0000 7f93b180f640 10 monclient: get_monmap_and_config 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.102+0000 7f93b180f640 10 monclient: build_initial_monmap 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.102+0000 7f93b180f640 10 monclient: monmap: 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:33.103144+0000 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:33.103144+0000 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:33.083 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.103+0000 7f93b180f640 1 Processor -- start 2024-03-29T10:56:33.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.103+0000 7f93b180f640 1 -- start start 2024-03-29T10:56:33.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.103+0000 7f93b180f640 10 monclient: init 2024-03-29T10:56:33.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.104+0000 7f93b180f640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:33.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.104+0000 7f93b180f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:33.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.104+0000 7f93b180f640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:33.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.105+0000 7f93b180f640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac13f650 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:33.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.105+0000 7f93aa7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f93ac13f650 0x7f93ac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58356/0 (socket says 172.21.15.64:58356) 2024-03-29T10:56:33.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.105+0000 7f93aa7fc640 1 -- 172.21.15.64:0/1077159108 learned_addr learned my addr 172.21.15.64:0/1077159108 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:33.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.105+0000 7f93b180f640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac0a9d70 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.105+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac0aaef0 con 0x7f93ac13f650 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac145bd0 con 0x7f93ac1457f0 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac145eb0 con 0x7f93ac0a9d70 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93b180f640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93b180f640 10 monclient(hunting): authenticate will time out at 1404.405209s 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917475622 0 0) 0x7f93ac145eb0 con 0x7f93ac0a9d70 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9394003520 con 0x7f93ac0a9d70 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650960186 0 0) 0x7f93ac0aaef0 con 0x7f93ac13f650 2024-03-29T10:56:33.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.106+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac145eb0 con 0x7f93ac13f650 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1365149111 0 0) 0x7f93ac145eb0 con 0x7f93ac13f650 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 >> v1:172.21.15.64:6790/0 conn(0x7f93ac0a9d70 legacy=0x7f93ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 >> v1:172.21.15.174:6789/0 conn(0x7f93ac1457f0 legacy=0x7f93ac13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 1 monclient: found mon.noname-a 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:33.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac146340 con 0x7f93ac13f650 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f93a0001670 con 0x7f93ac13f650 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93a9ffb640 10 monclient: dump: 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.093 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93b180f640 5 monclient: authenticate success, global_id 4578 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.107+0000 7f93b180f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93a00017d0 con 0x7f93ac13f650 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93a9ffb640 1 -- 172.21.15.64:0/1077159108 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f93a0001be0 con 0x7f93ac13f650 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93a9ffb640 10 monclient: dump: 2024-03-29T10:56:33.094 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93b180f640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.108+0000 7f93b180f640 4 set_mon_vals no callback set 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 10 monclient: shutdown 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 >> v1:172.21.15.64:6789/0 conn(0x7f93ac13f650 legacy=0x7f93ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:33.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 shutdown_connections 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 >> 172.21.15.64:0/1077159108 conn(0x7f93ac066a00 msgr2=0x7f93ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 shutdown_connections 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.109+0000 7f93b180f640 1 -- 172.21.15.64:0/1077159108 wait complete. 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.110+0000 7f93b180f640 10 monclient: build_initial_monmap 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.110+0000 7f93b180f640 10 monclient: monmap: 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:33.096 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:33.097 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.110+0000 7f93b180f640 1 librados: starting msgr at 2024-03-29T10:56:33.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.110+0000 7f93b180f640 1 librados: starting objecter 2024-03-29T10:56:33.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.111+0000 7f93b180f640 1 Processor -- start 2024-03-29T10:56:33.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.111+0000 7f93b180f640 1 -- start start 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.111+0000 7f93b180f640 1 librados: setting wanted keys 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.111+0000 7f93b180f640 1 librados: calling monclient init 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.111+0000 7f93b180f640 10 monclient: init 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac0a9d70 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac13f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac1457f0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac132950 con 0x7f93ac1457f0 2024-03-29T10:56:33.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac132c30 con 0x7f93ac0a9d70 2024-03-29T10:56:33.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac132f10 con 0x7f93ac13f650 2024-03-29T10:56:33.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:33.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93b180f640 10 monclient(hunting): authenticate will time out at 1404.411171s 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93aa7fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f93ac13f650 0x7f93ac142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37128/0 (socket says 172.21.15.64:37128) 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.112+0000 7f93aa7fc640 1 -- 172.21.15.64:0/2742723345 learned_addr learned my addr 172.21.15.64:0/2742723345 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 client.?.objecter ms_handle_connect 0x7f93ac13f650 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 client.?.objecter ms_handle_connect 0x7f93ac1457f0 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284649929 0 0) 0x7f93ac132f10 con 0x7f93ac13f650 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:33.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9398004eb0 con 0x7f93ac13f650 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620948912 0 0) 0x7f93ac132950 con 0x7f93ac1457f0 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 10 monclient(hunting): my global_id is 4581 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.113+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac132f10 con 0x7f93ac1457f0 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281911998 0 0) 0x7f93ac132f10 con 0x7f93ac1457f0 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93ac132950 con 0x7f93ac1457f0 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250496954 0 0) 0x7f9398004eb0 con 0x7f93ac13f650 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93ac132f10 con 0x7f93ac13f650 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f93a4001c50 con 0x7f93ac1457f0 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:33.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): dump: 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f93a0001be0 con 0x7f93ac13f650 2024-03-29T10:56:33.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 10 monclient(hunting): dump: 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4252864211 0 0) 0x7f93ac132f10 con 0x7f93ac13f650 2024-03-29T10:56:33.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.114+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 >> v1:172.21.15.174:6789/0 conn(0x7f93ac0a9d70 legacy=0x7f93ac133330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 >> v1:172.21.15.64:6789/0 conn(0x7f93ac1457f0 legacy=0x7f93ac1e2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 1 monclient: found mon.c 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac142250 con 0x7f93ac13f650 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 10 monclient: _finish_auth 0 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f9393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 5 monclient: authenticate success, global_id 4652 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 10 monclient: _renew_subs 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:33.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac140db0 con 0x7f93ac13f650 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 10 monclient: _renew_subs 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac13dfd0 con 0x7f93ac13f650 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.116+0000 7f93b180f640 1 librados: init done 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93a0004a90 con 0x7f93ac13f650 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f9393fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f93a0004e60 con 0x7f93ac13f650 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f9393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f9393fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f9393fff640 10 monclient: dump: 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:33.111 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.117+0000 7f93a97fa640 4 set_mon_vals no callback set 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.119+0000 7f93b180f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.119+0000 7f93b180f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.119+0000 7f93b180f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.119+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c005080 con 0x7f93ac13f650 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.119+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f93a0004fc0 con 0x7f93ac13f650 2024-03-29T10:56:33.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.120+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7f93a0084bf0 con 0x7f93ac13f650 2024-03-29T10:56:33.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.120+0000 7f9393fff640 10 client.4652.objecter ms_dispatch 0x7f93ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:33.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.120+0000 7f9393fff640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:33.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.120+0000 7f9393fff640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:33.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.120+0000 7f9393fff640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T10:56:33.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.122+0000 7f93aaffd640 10 monclient: get_auth_request con 0x7f939806df50 auth_method 0 2024-03-29T10:56:33.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.124+0000 7f9393fff640 10 client.4652.objecter ms_handle_connect 0x7f939806df50 2024-03-29T10:56:33.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.128+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93a00504d0 con 0x7f93ac13f650 2024-03-29T10:56:33.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.128+0000 7f9393fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:33.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.128+0000 7f9393fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:33.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.462+0000 7f93b180f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:33.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.462+0000 7f93b180f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:33.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.462+0000 7f93b180f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:33.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.462+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f938c005330 con 0x7f93ac13f650 2024-03-29T10:56:33.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.469+0000 7f9393fff640 1 -- 172.21.15.64:0/2742723345 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7f93a0052400 con 0x7f93ac13f650 2024-03-29T10:56:33.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.469+0000 7f9393fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:33.449 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.469+0000 7f9393fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:33.450 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:33.450 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:33.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 10 librados: watch_flush enter 2024-03-29T10:56:33.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 10 librados: watch_flush exit 2024-03-29T10:56:33.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T10:56:33.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 10 client.4652.objecter successfully canceled tick 2024-03-29T10:56:33.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 >> v1:172.21.15.174:6816/2473791532 conn(0x7f939806df50 legacy=0x7f9398070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:33.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 10 monclient: shutdown 2024-03-29T10:56:33.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:33.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.473+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 >> v1:172.21.15.64:6790/0 conn(0x7f93ac13f650 legacy=0x7f93ac142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:33.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.474+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 shutdown_connections 2024-03-29T10:56:33.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.474+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 >> 172.21.15.64:0/2742723345 conn(0x7f93ac066a00 msgr2=0x7f93ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:33.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.478+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 shutdown_connections 2024-03-29T10:56:33.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.478+0000 7f93b180f640 1 -- 172.21.15.64:0/2742723345 wait complete. 2024-03-29T10:56:33.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:33.478+0000 7f93b180f640 1 librados: shutdown 2024-03-29T10:56:33.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:40.284 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.468+0000 7f455bba3640 10 monclient: get_monmap_and_config 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.468+0000 7f455bba3640 10 monclient: build_initial_monmap 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.468+0000 7f455bba3640 10 monclient: monmap: 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:40.470310+0000 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:40.470310+0000 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:40.450 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.469+0000 7f455bba3640 1 Processor -- start 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.469+0000 7f455bba3640 1 -- start start 2024-03-29T10:56:40.451 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.469+0000 7f455bba3640 10 monclient: init 2024-03-29T10:56:40.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:40.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:40.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient(hunting): picked mon.noname-b con 0x7f45540b4330 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:40.457 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient(hunting): picked mon.noname-c con 0x7f45541470b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient(hunting): picked mon.noname-a con 0x7f45541439b0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455406f2a0 con 0x7f45541439b0 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4554146b90 con 0x7f45541470b0 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45540b2ca0 con 0x7f45540b4330 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.470+0000 7f455bba3640 10 monclient(hunting): authenticate will time out at 1411.770608s 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.471+0000 7f455913e640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f45541470b0 0x7f45540b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:44276/0 (socket says 172.21.15.64:44276) 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.471+0000 7f455913e640 1 -- 172.21.15.64:0/1136515790 learned_addr learned my addr 172.21.15.64:0/1136515790 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361414363 0 0) 0x7f4554146b90 con 0x7f45541470b0 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _negotiate 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:40.458 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): my global_id is 4600 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c003520 con 0x7f45541470b0 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834509435 0 0) 0x7f455406f2a0 con 0x7f45541439b0 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _negotiate 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554146b90 con 0x7f45541439b0 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977833480 0 0) 0x7f45540b2ca0 con 0x7f45540b4330 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.472+0000 7f455893d640 10 monclient(hunting): _negotiate 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:40.459 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455406f2a0 con 0x7f45540b4330 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2014707664 0 0) 0x7f453c003520 con 0x7f45541470b0 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 >> v1:172.21.15.64:6790/0 conn(0x7f45540b4330 legacy=0x7f45540b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 >> v1:172.21.15.64:6789/0 conn(0x7f45541439b0 legacy=0x7f4554151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient: _finish_hunting 0 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 monclient: found mon.noname-c 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.174:6789/0 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455406e0c0 con 0x7f45541470b0 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient: _finish_auth 0 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient: _check_auth_tickets 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:40.460 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455bba3640 5 monclient: authenticate success, global_id 4600 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455bba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f45480015c0 con 0x7f45541470b0 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.473+0000 7f455893d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 10 monclient: dump: 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:40.461 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455bba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.1 v1:172.21.15.174:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4548001720 con 0x7f45541470b0 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455bba3640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455bba3640 4 set_mon_vals no callback set 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 1 -- 172.21.15.64:0/1136515790 <== mon.1 v1:172.21.15.174:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4548001b30 con 0x7f45541470b0 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.474+0000 7f455893d640 10 monclient: dump: 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:40.462 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.475+0000 7f455bba3640 10 monclient: shutdown 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.475+0000 7f455bba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.475+0000 7f455bba3640 1 -- 172.21.15.64:0/1136515790 >> v1:172.21.15.174:6789/0 conn(0x7f45541470b0 legacy=0x7f45540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.475+0000 7f455bba3640 1 -- 172.21.15.64:0/1136515790 shutdown_connections 2024-03-29T10:56:40.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.475+0000 7f455bba3640 1 -- 172.21.15.64:0/1136515790 >> 172.21.15.64:0/1136515790 conn(0x7f4554066a00 msgr2=0x7f4554066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:40.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.476+0000 7f455bba3640 1 -- 172.21.15.64:0/1136515790 shutdown_connections 2024-03-29T10:56:40.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.476+0000 7f455bba3640 1 -- 172.21.15.64:0/1136515790 wait complete. 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.477+0000 7f455bba3640 10 monclient: build_initial_monmap 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.477+0000 7f455bba3640 10 monclient: monmap: 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:40.470 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.477+0000 7f455bba3640 1 librados: starting msgr at 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.477+0000 7f455bba3640 1 librados: starting objecter 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.478+0000 7f455bba3640 1 Processor -- start 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.478+0000 7f455bba3640 1 -- start start 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.478+0000 7f455bba3640 1 librados: setting wanted keys 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.478+0000 7f455bba3640 1 librados: calling monclient init 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.478+0000 7f455bba3640 10 monclient: init 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient(hunting): picked mon.noname-c con 0x7f45540b4330 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:40.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient(hunting): picked mon.noname-b con 0x7f45541439b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient(hunting): picked mon.noname-a con 0x7f45541df8c0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45541dfda0 con 0x7f45541df8c0 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4554141130 con 0x7f45541439b0 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45541412c0 con 0x7f45540b4330 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455bba3640 10 monclient(hunting): authenticate will time out at 1411.779554s 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455a140640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f45541df8c0 0x7f4554125640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43526/0 (socket says 172.21.15.64:43526) 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.479+0000 7f455a140640 1 -- 172.21.15.64:0/1894606343 learned_addr learned my addr 172.21.15.64:0/1894606343 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.480+0000 7f45527fc640 10 client.?.objecter ms_handle_connect 0x7f45540b4330 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.480+0000 7f45527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.480+0000 7f45527fc640 10 client.?.objecter ms_handle_connect 0x7f45541439b0 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.480+0000 7f45527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.480+0000 7f45527fc640 10 client.?.objecter ms_handle_connect 0x7f45541df8c0 2024-03-29T10:56:40.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.480+0000 7f45527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657855831 0 0) 0x7f4554141130 con 0x7f45541439b0 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): my global_id is 4603 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4540004eb0 con 0x7f45541439b0 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352472318 0 0) 0x7f45541dfda0 con 0x7f45541df8c0 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554141130 con 0x7f45541df8c0 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640946064 0 0) 0x7f45541412c0 con 0x7f45540b4330 2024-03-29T10:56:40.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45541dfda0 con 0x7f45540b4330 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250006195 0 0) 0x7f4540004eb0 con 0x7f45541439b0 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45541412c0 con 0x7f45541439b0 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.483+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4548001a30 con 0x7f45541439b0 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient(hunting): dump: 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:40.474 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266888341 0 0) 0x7f45541412c0 con 0x7f45541439b0 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 >> v1:172.21.15.64:6790/0 conn(0x7f45540b4330 legacy=0x7f45541df1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 >> v1:172.21.15.64:6789/0 conn(0x7f45541df8c0 legacy=0x7f4554125640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 1 monclient: found mon.b 2024-03-29T10:56:40.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554141450 con 0x7f45541439b0 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.484+0000 7f45527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f45527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 5 monclient: authenticate success, global_id 4603 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 10 monclient: _renew_subs 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4554140b30 con 0x7f45541439b0 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4548004a90 con 0x7f45541439b0 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f45527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:40.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f4553fff640 4 set_mon_vals no callback set 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 10 monclient: _renew_subs 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4554125d50 con 0x7f45541439b0 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.485+0000 7f455bba3640 1 librados: init done 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.486+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f4548004da0 con 0x7f45541439b0 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.486+0000 7f45527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.486+0000 7f45527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.486+0000 7f45527fc640 10 monclient: dump: 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:40.477 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.487+0000 7f455bba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.487+0000 7f455bba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.487+0000 7f455bba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.487+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4554127b50 con 0x7f45541439b0 2024-03-29T10:56:40.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.488+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f454801ad70 con 0x7f45541439b0 2024-03-29T10:56:40.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f455993f640 10 monclient: get_auth_request con 0x7f454006de80 auth_method 0 2024-03-29T10:56:40.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7f4548084a70 con 0x7f45541439b0 2024-03-29T10:56:40.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f45527fc640 10 client.4603.objecter ms_dispatch 0x7f4554000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f45527fc640 3 client.4603.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f45527fc640 3 client.4603.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f45527fc640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.489+0000 7f45527fc640 10 client.4603.objecter ms_handle_connect 0x7f454006de80 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.495+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4548050180 con 0x7f45541439b0 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.495+0000 7f45527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:40.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.495+0000 7f45527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:40.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.894+0000 7f455bba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:40.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.894+0000 7f455bba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:40.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.894+0000 7f455bba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.174:6789/0 2024-03-29T10:56:40.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.894+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 --> v1:172.21.15.174:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4554071910 con 0x7f45541439b0 2024-03-29T10:56:40.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.896+0000 7f45527fc640 1 -- 172.21.15.64:0/1894606343 <== mon.1 v1:172.21.15.174:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7f4548052280 con 0x7f45541439b0 2024-03-29T10:56:40.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.896+0000 7f45527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:40.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.896+0000 7f45527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:40.878 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:40.878 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:40.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 10 librados: watch_flush enter 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 10 librados: watch_flush exit 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 10 client.4603.objecter successfully canceled tick 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 >> v1:172.21.15.174:6816/2473791532 conn(0x7f454006de80 legacy=0x7f4540070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 10 monclient: shutdown 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:40.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 >> v1:172.21.15.174:6789/0 conn(0x7f45541439b0 legacy=0x7f4554140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:40.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 shutdown_connections 2024-03-29T10:56:40.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.899+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 >> 172.21.15.64:0/1894606343 conn(0x7f4554066a00 msgr2=0x7f4554145de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:40.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.900+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 shutdown_connections 2024-03-29T10:56:40.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.900+0000 7f455bba3640 1 -- 172.21.15.64:0/1894606343 wait complete. 2024-03-29T10:56:40.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:40.900+0000 7f455bba3640 1 librados: shutdown 2024-03-29T10:56:40.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:47.702 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.883+0000 7f81ce4e2640 10 monclient: get_monmap_and_config 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.883+0000 7f81ce4e2640 10 monclient: build_initial_monmap 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.884+0000 7f81ce4e2640 10 monclient: monmap: 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:47.884813+0000 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:47.884813+0000 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:47.866 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:47.867 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:47.867 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:47.867 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:47.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.886+0000 7f81ce4e2640 1 Processor -- start 2024-03-29T10:56:47.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.889+0000 7f81ce4e2640 1 -- start start 2024-03-29T10:56:47.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.889+0000 7f81ce4e2640 10 monclient: init 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient(hunting): picked mon.noname-c con 0x7f81c81422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient(hunting): picked mon.noname-b con 0x7f81c8142690 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient(hunting): picked mon.noname-a con 0x7f81c80b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8142c80 con 0x7f81c80b17d0 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80aae40 con 0x7f81c81422b0 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c813fbd0 con 0x7f81c8142690 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:47.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.890+0000 7f81ce4e2640 10 monclient(hunting): authenticate will time out at 1419.189579s 2024-03-29T10:56:47.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.893+0000 7f81c7fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f81c81422b0 0x7f81c813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.64:36465/0 (socket says 172.21.15.64:36465) 2024-03-29T10:56:47.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.893+0000 7f81c7fff640 1 -- 172.21.15.64:0/2771933210 learned_addr learned my addr 172.21.15.64:0/2771933210 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732761991 0 0) 0x7f81c813fbd0 con 0x7f81c8142690 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0003520 con 0x7f81c8142690 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165448702 0 0) 0x7f81c80aae40 con 0x7f81c81422b0 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:47.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 10 monclient(hunting): my global_id is 4606 2024-03-29T10:56:47.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b00048a0 con 0x7f81c81422b0 2024-03-29T10:56:47.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27803032 0 0) 0x7f81b0003520 con 0x7f81c8142690 2024-03-29T10:56:47.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 >> v1:172.21.15.174:6789/0 conn(0x7f81c81422b0 legacy=0x7f81c813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:47.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.894+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 >> v1:172.21.15.64:6789/0 conn(0x7f81c80b17d0 legacy=0x7f81c80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c8151940 con 0x7f81c8142690 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f81bc001670 con 0x7f81c8142690 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:47.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81c6ffd640 10 monclient: dump: 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81ce4e2640 5 monclient: authenticate success, global_id 4661 2024-03-29T10:56:47.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.897+0000 7f81ce4e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:47.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81bc001850 con 0x7f81c8142690 2024-03-29T10:56:47.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81c6ffd640 1 -- 172.21.15.64:0/2771933210 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f81bc001be0 con 0x7f81c8142690 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81c6ffd640 10 monclient: dump: 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:47.882 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:47.883 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:47.883 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:47.883 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:47.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81ce4e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:47.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.901+0000 7f81ce4e2640 4 set_mon_vals no callback set 2024-03-29T10:56:47.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.902+0000 7f81ce4e2640 10 monclient: shutdown 2024-03-29T10:56:47.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.902+0000 7f81ce4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:47.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.902+0000 7f81ce4e2640 1 -- 172.21.15.64:0/2771933210 >> v1:172.21.15.64:6790/0 conn(0x7f81c8142690 legacy=0x7f81c80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:47.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.905+0000 7f81ce4e2640 1 -- 172.21.15.64:0/2771933210 shutdown_connections 2024-03-29T10:56:47.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.905+0000 7f81ce4e2640 1 -- 172.21.15.64:0/2771933210 >> 172.21.15.64:0/2771933210 conn(0x7f81c8066a00 msgr2=0x7f81c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:47.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.905+0000 7f81ce4e2640 1 -- 172.21.15.64:0/2771933210 shutdown_connections 2024-03-29T10:56:47.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.905+0000 7f81ce4e2640 1 -- 172.21.15.64:0/2771933210 wait complete. 2024-03-29T10:56:47.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.907+0000 7f81ce4e2640 10 monclient: build_initial_monmap 2024-03-29T10:56:47.887 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.907+0000 7f81ce4e2640 10 monclient: monmap: 2024-03-29T10:56:47.887 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:47.887 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:47.887 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.907+0000 7f81ce4e2640 1 librados: starting msgr at 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.907+0000 7f81ce4e2640 1 librados: starting objecter 2024-03-29T10:56:47.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.908+0000 7f81ce4e2640 1 Processor -- start 2024-03-29T10:56:47.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.909+0000 7f81ce4e2640 1 -- start start 2024-03-29T10:56:47.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.909+0000 7f81ce4e2640 1 librados: setting wanted keys 2024-03-29T10:56:47.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.909+0000 7f81ce4e2640 1 librados: calling monclient init 2024-03-29T10:56:47.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.909+0000 7f81ce4e2640 10 monclient: init 2024-03-29T10:56:47.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.911+0000 7f81ce4e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:47.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.912+0000 7f81ce4e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:47.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.912+0000 7f81ce4e2640 10 monclient(hunting): picked mon.noname-a con 0x7f81c80b17d0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:47.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 10 monclient(hunting): picked mon.noname-b con 0x7f81c81422b0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:47.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 10 monclient(hunting): picked mon.noname-c con 0x7f81c8142690 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:47.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8130fb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8130cd0 con 0x7f81c81422b0 2024-03-29T10:56:47.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80ac9a0 con 0x7f81c8142690 2024-03-29T10:56:47.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:47.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81ce4e2640 10 monclient(hunting): authenticate will time out at 1419.213071s 2024-03-29T10:56:47.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81cca7f640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f81c8142690 0x7f81c81e2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38034/0 (socket says 172.21.15.64:38034) 2024-03-29T10:56:47.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.914+0000 7f81cca7f640 1 -- 172.21.15.64:0/547767142 learned_addr learned my addr 172.21.15.64:0/547767142 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:47.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 client.?.objecter ms_handle_connect 0x7f81c8142690 2024-03-29T10:56:47.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:47.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 client.?.objecter ms_handle_connect 0x7f81c80b17d0 2024-03-29T10:56:47.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:47.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325052432 0 0) 0x7f81c8130fb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.916+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b4004eb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548443917 0 0) 0x7f81b4004eb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8130fb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f81c0001c30 con 0x7f81c80b17d0 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:47.900 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:47.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1810101124 0 0) 0x7f81c8130fb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 >> v1:172.21.15.64:6790/0 conn(0x7f81c8142690 legacy=0x7f81c81e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.919+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 >> v1:172.21.15.174:6789/0 conn(0x7f81c81422b0 legacy=0x7f81c812ec20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 1 monclient: found mon.a 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:47.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c80acc10 con 0x7f81c80b17d0 2024-03-29T10:56:47.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:47.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:47.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.920+0000 7f81c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 5 monclient: authenticate success, global_id 4590 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 10 monclient: _renew_subs 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c80acef0 con 0x7f81c80b17d0 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 10 monclient: _renew_subs 2024-03-29T10:56:47.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:47.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c80abeb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.923+0000 7f81ce4e2640 1 librados: init done 2024-03-29T10:56:47.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.925+0000 7f81ce4e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:47.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.925+0000 7f81ce4e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:47.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.925+0000 7f81ce4e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:47.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.926+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c80ad410 con 0x7f81c80b17d0 2024-03-29T10:56:47.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.929+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c0004b60 con 0x7f81c80b17d0 2024-03-29T10:56:47.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.929+0000 7f81c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:47.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.929+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f81c0004d90 con 0x7f81c80b17d0 2024-03-29T10:56:47.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.929+0000 7f81c5ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.930+0000 7f81c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.931+0000 7f81c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.931+0000 7f81c4ff9640 10 monclient: dump: 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.911 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:47.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.931+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f81c0004fb0 con 0x7f81c80b17d0 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7f81c0084d60 con 0x7f81c80b17d0 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c4ff9640 10 client.4590.objecter ms_dispatch 0x7f81c8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c77fe640 10 monclient: get_auth_request con 0x7f81b406dd90 auth_method 0 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c4ff9640 3 client.4590.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c4ff9640 3 client.4590.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c4ff9640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-29T10:56:47.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.932+0000 7f81c4ff9640 10 client.4590.objecter ms_handle_connect 0x7f81b406dd90 2024-03-29T10:56:47.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.935+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81c00506c0 con 0x7f81c80b17d0 2024-03-29T10:56:47.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.936+0000 7f81c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:47.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:47.936+0000 7f81c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:48.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.349+0000 7f81ce4e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:48.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.349+0000 7f81ce4e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:48.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.349+0000 7f81ce4e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:56:48.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.349+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81c8075f10 con 0x7f81c80b17d0 2024-03-29T10:56:48.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.350+0000 7f81c4ff9640 1 -- 172.21.15.64:0/547767142 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7f81c00525f0 con 0x7f81c80b17d0 2024-03-29T10:56:48.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.350+0000 7f81c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:48.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.350+0000 7f81c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:48.331 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:48.331 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:48.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 10 librados: watch_flush enter 2024-03-29T10:56:48.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 10 librados: watch_flush exit 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 10 client.4590.objecter successfully canceled tick 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 >> v1:172.21.15.174:6816/2473791532 conn(0x7f81b406dd90 legacy=0x7f81b4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 10 monclient: shutdown 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 >> v1:172.21.15.64:6789/0 conn(0x7f81c80b17d0 legacy=0x7f81c81323a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.354+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 shutdown_connections 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.355+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 >> 172.21.15.64:0/547767142 conn(0x7f81c8066a00 msgr2=0x7f81c80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.355+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 shutdown_connections 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.355+0000 7f81ce4e2640 1 -- 172.21.15.64:0/547767142 wait complete. 2024-03-29T10:56:48.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:48.355+0000 7f81ce4e2640 1 librados: shutdown 2024-03-29T10:56:48.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:55.157 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:55.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.322+0000 7fbfe6e9a640 10 monclient: get_monmap_and_config 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.322+0000 7fbfe6e9a640 10 monclient: build_initial_monmap 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.322+0000 7fbfe6e9a640 10 monclient: monmap: 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:56:55.322927+0000 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:56:55.322927+0000 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:56:55.303 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:55.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.326+0000 7fbfe6e9a640 1 Processor -- start 2024-03-29T10:56:55.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.326+0000 7fbfe6e9a640 1 -- start start 2024-03-29T10:56:55.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.326+0000 7fbfe6e9a640 10 monclient: init 2024-03-29T10:56:55.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.327+0000 7fbfe6e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:55.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.327+0000 7fbfe6e9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:55.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe00a9750 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:55.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe013e1e0 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:55.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe0071c30 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00ab300 con 0x7fbfe0071c30 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00ac2f0 con 0x7fbfe013e1e0 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00ad8d0 con 0x7fbfe00a9750 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.330+0000 7fbfe6e9a640 10 monclient(hunting): authenticate will time out at 1426.629786s 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.331+0000 7fbfe4c36640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbfe00a9750 0x7fbfe013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37161/0 (socket says 172.21.15.64:37161) 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.331+0000 7fbfe4c36640 1 -- 172.21.15.64:0/1843842030 learned_addr learned my addr 172.21.15.64:0/1843842030 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:55.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429647909 0 0) 0x7fbfe00ad8d0 con 0x7fbfe00a9750 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003520 con 0x7fbfe00a9750 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539823691 0 0) 0x7fbfc8003520 con 0x7fbfe00a9750 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.332+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 >> v1:172.21.15.174:6789/0 conn(0x7fbfe013e1e0 legacy=0x7fbfe00ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:55.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.333+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 >> v1:172.21.15.64:6789/0 conn(0x7fbfe0071c30 legacy=0x7fbfe00ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe00ada60 con 0x7fbfe00a9750 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:55.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfe6e9a640 5 monclient: authenticate success, global_id 4667 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfe6e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbfd0001690 con 0x7fbfe00a9750 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: dump: 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:55.321 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfe6e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfd0001920 con 0x7fbfe00a9750 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.334+0000 7fbfe6e9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.335+0000 7fbfe6e9a640 4 set_mon_vals no callback set 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.335+0000 7fbfe6e9a640 10 monclient: shutdown 2024-03-29T10:56:55.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.335+0000 7fbfe6e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.336+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/1843842030 >> v1:172.21.15.64:6790/0 conn(0x7fbfe00a9750 legacy=0x7fbfe013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.336+0000 7fbfdf7fe640 1 -- 172.21.15.64:0/1843842030 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbfd0001cf0 con 0x7fbfe00a9750 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.336+0000 7fbfdf7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.336+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/1843842030 shutdown_connections 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.336+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/1843842030 >> 172.21.15.64:0/1843842030 conn(0x7fbfe006b660 msgr2=0x7fbfe006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.337+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/1843842030 shutdown_connections 2024-03-29T10:56:55.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.337+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/1843842030 wait complete. 2024-03-29T10:56:55.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.338+0000 7fbfe6e9a640 10 monclient: build_initial_monmap 2024-03-29T10:56:55.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.338+0000 7fbfe6e9a640 10 monclient: monmap: 2024-03-29T10:56:55.327 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:56:55.327 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:55.327 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:56:55.327 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 1 librados: starting msgr at 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 1 librados: starting objecter 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 1 Processor -- start 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 1 -- start start 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 1 librados: setting wanted keys 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 1 librados: calling monclient init 2024-03-29T10:56:55.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.339+0000 7fbfe6e9a640 10 monclient: init 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.341+0000 7fbfe6e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.341+0000 7fbfe6e9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe0071c30 addr v1:172.21.15.64:6790/0 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe00a9750 addr v1:172.21.15.174:6789/0 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe013e1e0 addr v1:172.21.15.64:6789/0 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe0122840 con 0x7fbfe013e1e0 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe0123190 con 0x7fbfe00a9750 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe0122a90 con 0x7fbfe0071c30 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.342+0000 7fbfe6e9a640 10 monclient(hunting): authenticate will time out at 1426.641377s 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.344+0000 7fbfe4c36640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbfe0071c30 0x7fbfe01213b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37273/0 (socket says 172.21.15.64:37273) 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.344+0000 7fbfe4c36640 1 -- 172.21.15.64:0/713018124 learned_addr learned my addr 172.21.15.64:0/713018124 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:56:55.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.344+0000 7fbfdd7fa640 10 client.?.objecter ms_handle_connect 0x7fbfe0071c30 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.344+0000 7fbfdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.344+0000 7fbfdd7fa640 10 client.?.objecter ms_handle_connect 0x7fbfe013e1e0 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.344+0000 7fbfdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.346+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904527783 0 0) 0x7fbfe0122a90 con 0x7fbfe0071c30 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.346+0000 7fbfdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.346+0000 7fbfdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.346+0000 7fbfdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.346+0000 7fbfdd7fa640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:56:55.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.346+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfcc004eb0 con 0x7fbfe0071c30 2024-03-29T10:56:55.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905409073 0 0) 0x7fbfcc004eb0 con 0x7fbfe0071c30 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe0122a90 con 0x7fbfe0071c30 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887213719 0 0) 0x7fbfe0122840 con 0x7fbfe013e1e0 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfcc004eb0 con 0x7fbfe013e1e0 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbfd0001610 con 0x7fbfe0071c30 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:55.335 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.347+0000 7fbfdd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2945823642 0 0) 0x7fbfe0122a90 con 0x7fbfe0071c30 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 >> v1:172.21.15.174:6789/0 conn(0x7fbfe00a9750 legacy=0x7fbfe00b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 >> v1:172.21.15.64:6789/0 conn(0x7fbfe013e1e0 legacy=0x7fbfe01e01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:55.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 1 monclient: found mon.c 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.348+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0122d70 con 0x7fbfe0071c30 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfd0004c30 con 0x7fbfe0071c30 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7fbfd0004f40 con 0x7fbfe0071c30 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdd7fa640 10 monclient: dump: 2024-03-29T10:56:55.337 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 5 monclient: authenticate success, global_id 4670 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfdeffd640 4 set_mon_vals no callback set 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 10 monclient: _renew_subs 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:55.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe0123410 con 0x7fbfe0071c30 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 10 monclient: _renew_subs 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe0121ac0 con 0x7fbfe0071c30 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.349+0000 7fbfe6e9a640 1 librados: init done 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.351+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7fbfd001af10 con 0x7fbfe0071c30 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.352+0000 7fbfc6ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.352+0000 7fbfc6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.352+0000 7fbfc6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.352+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb8005080 con 0x7fbfe0071c30 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.352+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7fbfd0084ac0 con 0x7fbfe0071c30 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.353+0000 7fbfdd7fa640 10 client.4670.objecter ms_dispatch 0x7fbfe0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.353+0000 7fbfdd7fa640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:56:55.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.353+0000 7fbfdd7fa640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:56:55.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.353+0000 7fbfdd7fa640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T10:56:55.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.353+0000 7fbfdffff640 10 monclient: get_auth_request con 0x7fbfcc06dce0 auth_method 0 2024-03-29T10:56:55.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.353+0000 7fbfdd7fa640 10 client.4670.objecter ms_handle_connect 0x7fbfcc06dce0 2024-03-29T10:56:55.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.361+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfd00503a0 con 0x7fbfe0071c30 2024-03-29T10:56:55.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.361+0000 7fbfdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:55.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.361+0000 7fbfdd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:55.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.697+0000 7fbfc6ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:55.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.697+0000 7fbfc6ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:55.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.697+0000 7fbfc6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-29T10:56:55.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.697+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb8005330 con 0x7fbfe0071c30 2024-03-29T10:56:55.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.698+0000 7fbfdd7fa640 1 -- 172.21.15.64:0/713018124 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7fbfd0004da0 con 0x7fbfe0071c30 2024-03-29T10:56:55.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.698+0000 7fbfdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:55.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.698+0000 7fbfdd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:55.679 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:56:55.679 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:55.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.700+0000 7fbfc6ffd640 10 librados: watch_flush enter 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.700+0000 7fbfc6ffd640 10 librados: watch_flush exit 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.700+0000 7fbfc6ffd640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 10 client.4670.objecter successfully canceled tick 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 >> v1:172.21.15.174:6816/2473791532 conn(0x7fbfcc06dce0 legacy=0x7fbfcc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 10 monclient: shutdown 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 >> v1:172.21.15.64:6790/0 conn(0x7fbfe0071c30 legacy=0x7fbfe01213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 shutdown_connections 2024-03-29T10:56:55.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.701+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 >> 172.21.15.64:0/713018124 conn(0x7fbfe006b660 msgr2=0x7fbfe00b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:55.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.702+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 shutdown_connections 2024-03-29T10:56:55.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.702+0000 7fbfc6ffd640 1 -- 172.21.15.64:0/713018124 wait complete. 2024-03-29T10:56:55.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:56:55.702+0000 7fbfc6ffd640 1 librados: shutdown 2024-03-29T10:56:55.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:02.506 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.679+0000 7f28a1c74640 10 monclient: get_monmap_and_config 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.679+0000 7f28a1c74640 10 monclient: build_initial_monmap 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.679+0000 7f28a1c74640 10 monclient: monmap: 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:57:02.680425+0000 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:57:02.680425+0000 2024-03-29T10:57:02.660 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-c 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.680+0000 7f28a1c74640 1 Processor -- start 2024-03-29T10:57:02.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.681+0000 7f28a1c74640 1 -- start start 2024-03-29T10:57:02.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.681+0000 7f28a1c74640 10 monclient: init 2024-03-29T10:57:02.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:02.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient(hunting): picked mon.noname-c con 0x7f289c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient(hunting): picked mon.noname-b con 0x7f289c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient(hunting): picked mon.noname-a con 0x7f289c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c0aaef0 con 0x7f289c0a9d70 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c145bd0 con 0x7f289c1457f0 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c145eb0 con 0x7f289c13f650 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:02.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.682+0000 7f28a1c74640 10 monclient(hunting): authenticate will time out at 1433.981251s 2024-03-29T10:57:02.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289affd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f289c13f650 0x7f289c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40748/0 (socket says 172.21.15.64:40748) 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289affd640 1 -- 172.21.15.64:0/3346136187 learned_addr learned my addr 172.21.15.64:0/3346136187 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304317256 0 0) 0x7f289c145eb0 con 0x7f289c13f650 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289a7fc640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:57:02.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.683+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2884003520 con 0x7f289c13f650 2024-03-29T10:57:02.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460753371 0 0) 0x7f2884003520 con 0x7f289c13f650 2024-03-29T10:57:02.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 >> v1:172.21.15.174:6789/0 conn(0x7f289c1457f0 legacy=0x7f289c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:02.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 >> v1:172.21.15.64:6789/0 conn(0x7f289c0a9d70 legacy=0x7f289c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:02.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c146340 con 0x7f289c13f650 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f28a1c74640 5 monclient: authenticate success, global_id 4673 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f28a1c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f288c001690 con 0x7f289c13f650 2024-03-29T10:57:02.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: dump: 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:57:02.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f28a1c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:02.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f288c0018f0 con 0x7f289c13f650 2024-03-29T10:57:02.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f289a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f28a1c74640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.686+0000 7f28a1c74640 4 set_mon_vals no callback set 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f28a1c74640 10 monclient: shutdown 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f28a1c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f28a1c74640 1 -- 172.21.15.64:0/3346136187 >> v1:172.21.15.64:6790/0 conn(0x7f289c13f650 legacy=0x7f289c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f289a7fc640 1 -- 172.21.15.64:0/3346136187 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f288c001690 con 0x7f289c13f650 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f289a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f28a1c74640 1 -- 172.21.15.64:0/3346136187 shutdown_connections 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.687+0000 7f28a1c74640 1 -- 172.21.15.64:0/3346136187 >> 172.21.15.64:0/3346136187 conn(0x7f289c066a00 msgr2=0x7f289c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.688+0000 7f28a1c74640 1 -- 172.21.15.64:0/3346136187 shutdown_connections 2024-03-29T10:57:02.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.689+0000 7f28a1c74640 1 -- 172.21.15.64:0/3346136187 wait complete. 2024-03-29T10:57:02.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.690+0000 7f28a1c74640 10 monclient: build_initial_monmap 2024-03-29T10:57:02.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.690+0000 7f28a1c74640 10 monclient: monmap: 2024-03-29T10:57:02.671 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.noname-b 2024-03-29T10:57:02.672 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-29T10:57:02.673 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:57:02.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.692+0000 7f28a1c74640 1 librados: starting msgr at 2024-03-29T10:57:02.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.692+0000 7f28a1c74640 1 librados: starting objecter 2024-03-29T10:57:02.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.692+0000 7f28a1c74640 1 Processor -- start 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.692+0000 7f28a1c74640 1 -- start start 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.693+0000 7f28a1c74640 1 librados: setting wanted keys 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.693+0000 7f28a1c74640 1 librados: calling monclient init 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.693+0000 7f28a1c74640 10 monclient: init 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.693+0000 7f28a1c74640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.693+0000 7f28a1c74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:02.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.694+0000 7f28a1c74640 10 monclient(hunting): picked mon.noname-a con 0x7f289c0a9d70 addr v1:172.21.15.64:6789/0 2024-03-29T10:57:02.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.694+0000 7f28a1c74640 10 monclient(hunting): picked mon.noname-c con 0x7f289c13f650 addr v1:172.21.15.64:6790/0 2024-03-29T10:57:02.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.696+0000 7f289affd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f289c13f650 0x7f289c138a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:50278/0 (socket says 172.21.15.64:50278) 2024-03-29T10:57:02.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f289affd640 1 -- 172.21.15.64:0/4067631125 learned_addr learned my addr 172.21.15.64:0/4067631125 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-29T10:57:02.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f28a1c74640 10 monclient(hunting): picked mon.noname-b con 0x7f289c1457f0 addr v1:172.21.15.174:6789/0 2024-03-29T10:57:02.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c1ccc20 con 0x7f289c0a9d70 2024-03-29T10:57:02.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c1ccf00 con 0x7f289c1457f0 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 10 client.?.objecter ms_handle_connect 0x7f289c13f650 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c139c40 con 0x7f289c13f650 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f28a1c74640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f28a1c74640 10 monclient(hunting): authenticate will time out at 1433.996202s 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 10 client.?.objecter ms_handle_connect 0x7f289c0a9d70 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189088141 0 0) 0x7f289c1ccc20 con 0x7f289c0a9d70 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.697+0000 7f2883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:02.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888004eb0 con 0x7f289c0a9d70 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725195788 0 0) 0x7f289c139c40 con 0x7f289c13f650 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): my global_id is 4676 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289c1ccc20 con 0x7f289c13f650 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084307062 0 0) 0x7f2888004eb0 con 0x7f289c0a9d70 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f289c139c40 con 0x7f289c0a9d70 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f28900017d0 con 0x7f289c0a9d70 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.698+0000 7f2883fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:02.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): dump: 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:57:02.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 client.?.objecter ms_handle_connect 0x7f289c1457f0 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895678210 0 0) 0x7f289c1ccf00 con 0x7f289c1457f0 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 10 monclient(hunting): my global_id is 4609 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888004eb0 con 0x7f289c1457f0 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1673725682 0 0) 0x7f289c139c40 con 0x7f289c0a9d70 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 >> v1:172.21.15.64:6790/0 conn(0x7f289c13f650 legacy=0x7f289c138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.699+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 >> v1:172.21.15.174:6789/0 conn(0x7f289c1457f0 legacy=0x7f289c1391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:02.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 1 monclient: found mon.a 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c13b610 con 0x7f289c0a9d70 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2890001ce0 con 0x7f289c0a9d70 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2899ffb640 4 set_mon_vals no callback set 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1437008817 0 0) 0x7f2890003790 con 0x7f289c0a9d70 2024-03-29T10:57:02.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f2883fff640 10 monclient: dump: 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:fsid 0263dc76-f748-4029-877b-c4b256090311 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-29T10:45:26.324322+0000 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-29T10:45:26.324322+0000 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.174:6789/0 mon.b 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-29T10:57:02.694 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f28a1c74640 5 monclient: authenticate success, global_id 4599 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f28a1c74640 10 monclient: _renew_subs 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f28a1c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.700+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c1398b0 con 0x7f289c0a9d70 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.701+0000 7f28a1c74640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.701+0000 7f28a1c74640 10 monclient: _renew_subs 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.701+0000 7f28a1c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.701+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c1cd570 con 0x7f289c0a9d70 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.701+0000 7f28a1c74640 1 librados: init done 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.703+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4007816970 0 0) 0x7f28900038f0 con 0x7f289c0a9d70 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.704+0000 7f289affd640 10 monclient: get_auth_request con 0x7f288806de30 auth_method 0 2024-03-29T10:57:02.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.704+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3353452503 0 0) 0x7f2890083780 con 0x7f289c0a9d70 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.704+0000 7f2883fff640 10 client.4599.objecter ms_dispatch 0x7f289c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.704+0000 7f2883fff640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.704+0000 7f2883fff640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.704+0000 7f2883fff640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.705+0000 7f2883fff640 10 client.4599.objecter ms_handle_connect 0x7f288806de30 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.706+0000 7f28a1c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.706+0000 7f28a1c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.706+0000 7f28a1c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:57:02.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.706+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289c139dd0 con 0x7f289c0a9d70 2024-03-29T10:57:02.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.714+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f289004f060 con 0x7f289c0a9d70 2024-03-29T10:57:02.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.714+0000 7f2883fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:02.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:02.714+0000 7f2883fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:03.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.043+0000 7f28a1c74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:03.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.043+0000 7f28a1c74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:03.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.043+0000 7f28a1c74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-29T10:57:03.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.043+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f289c139a40 con 0x7f289c0a9d70 2024-03-29T10:57:03.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.048+0000 7f2883fff640 1 -- 172.21.15.64:0/4067631125 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1460183327) 0x7f2890050f90 con 0x7f289c0a9d70 2024-03-29T10:57:03.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.048+0000 7f2883fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:03.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.048+0000 7f2883fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:03.029 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-29T10:57:03.029 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"0263dc76-f748-4029-877b-c4b256090311","created":"2024-03-29T10:46:04.437210+0000","modified":"2024-03-29T10:56:09.373487+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:56:09.373487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29d8829f-a8b5-4888-a248-7d3f3ad95394","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b58ab0d3-43de-4f0c-b2fb-a6a557729219","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ab146959-79eb-4128-b51c-c83a6258a4fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"12211654-354e-4112-8f60-e176e2942e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"755189f4-ae84-4a5f-84c4-0fa8d3d9b0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42e46429-40bb-49c0-83a3-74d0047b0f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"03e1d62c-2309-4141-971a-b6f48b5dd70c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d61a9d8a-d7d8-4388-9738-50bce2ba2326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:03.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.051+0000 7f28a1c74640 10 librados: watch_flush enter 2024-03-29T10:57:03.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.054+0000 7f28a1c74640 10 librados: watch_flush exit 2024-03-29T10:57:03.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.054+0000 7f28a1c74640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T10:57:03.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.054+0000 7f28a1c74640 10 client.4599.objecter successfully canceled tick 2024-03-29T10:57:03.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 >> v1:172.21.15.174:6816/2473791532 conn(0x7f288806de30 legacy=0x7f28880702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 10 monclient: shutdown 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 >> v1:172.21.15.64:6789/0 conn(0x7f289c0a9d70 legacy=0x7f289c13af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 shutdown_connections 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 >> 172.21.15.64:0/4067631125 conn(0x7f289c066a00 msgr2=0x7f289c0b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.055+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 shutdown_connections 2024-03-29T10:57:03.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.056+0000 7f28a1c74640 1 -- 172.21.15.64:0/4067631125 wait complete. 2024-03-29T10:57:03.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-29T10:57:03.058+0000 7f28a1c74640 1 librados: shutdown 2024-03-29T10:57:03.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:03.056 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:57:03.058 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:57:03.058 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:57:03.058 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.105 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:57:03.105 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:57:03.105 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.144 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:57:03.145 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:57:03.145 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.217 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:57:03.217 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:57:03.217 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.286 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:57:03.286 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:57:03.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.328 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:57:03.328 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:57:03.328 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.363 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:57:03.364 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:57:03.364 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.436 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:57:03.436 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:57:03.436 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.504 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:57:03.505 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:57:03.505 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:57:03.505 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.528 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:57:03.529 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:57:03.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.570 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:57:03.570 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:57:03.570 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:57:03.570 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.581 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:57:03.582 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:57:03.582 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.633 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:57:03.633 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:57:03.634 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:57:03.645 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:57:03.645 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:57:03.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:57:03.706 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:57:03.707 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:57:06.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:57:06.545 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:57:06.675 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:57:06.675 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:57:06.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:57:06.742 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:57:06.822 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:57:06.822 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:57:09.486 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:57:09.486 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:57:09.565 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:57:09.565 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:57:09.630 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:57:09.630 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:57:09.703 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:57:09.705 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/data/mon.a.tgz 2024-03-29T10:57:09.705 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-29T10:57:09.723 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.35XF8KnLXp 2024-03-29T10:57:09.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.35XF8KnLXp 2024-03-29T10:57:09.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.35XF8KnLXp 2024-03-29T10:57:09.918 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.35XF8KnLXp is 126KB 2024-03-29T10:57:09.971 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.35XF8KnLXp 2024-03-29T10:57:09.986 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/data/mon.c.tgz 2024-03-29T10:57:09.986 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-29T10:57:10.041 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.HJdpAJpetT 2024-03-29T10:57:10.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.HJdpAJpetT 2024-03-29T10:57:10.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.HJdpAJpetT 2024-03-29T10:57:10.215 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.HJdpAJpetT is 126KB 2024-03-29T10:57:10.265 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.HJdpAJpetT 2024-03-29T10:57:10.279 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/data/mon.b.tgz 2024-03-29T10:57:10.279 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-03-29T10:57:10.297 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.qZLAfMdUCm 2024-03-29T10:57:10.298 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qZLAfMdUCm 2024-03-29T10:57:10.399 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.qZLAfMdUCm 2024-03-29T10:57:10.486 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.qZLAfMdUCm is 126KB 2024-03-29T10:57:10.539 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.qZLAfMdUCm 2024-03-29T10:57:10.554 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:57:10.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:57:10.558 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:57:10.637 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:57:10.675 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:57:10.675 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:57:10.677 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/remote/smithi064/crash 2024-03-29T10:57:10.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:57:10.704 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/remote/smithi174/crash 2024-03-29T10:57:10.705 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:57:10.741 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:57:10.742 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:57:10.748 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39361.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41760.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35102.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39361.log.gz 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41760.log.gz 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35102.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40259.log 2024-03-29T10:57:10.812 INFO:teuthology.orchestra.run.smithi064.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35102.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39752.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39166.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40259.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38971.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39752.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41058.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39166.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40181.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38971.log.gz 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-03-29T10:57:10.813 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41058.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38776.log 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40181.log: /var/log/ceph/ceph-client.admin.38815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40665.log 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40181.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41526.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38776.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40745.log 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34811.log 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40745.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40665.log.gz 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-03-29T10:57:10.814 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34811.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40745.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41448.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.34811.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38423.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39908.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41448.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38501.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40220.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38501.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz 2024-03-29T10:57:10.815 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39283.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.40220.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38501.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41292.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39322.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39283.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41292.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39322.log.gz 2024-03-29T10:57:10.816 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40454.log: gzip 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz -5 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40376.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41097.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40376.log.gz 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41097.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35205.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41097.log.gz 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41487.log 2024-03-29T10:57:10.817 INFO:teuthology.orchestra.run.smithi064.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41838.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35205.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41487.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41019.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41487.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41838.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41175.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41019.log.gz 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 85.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz /var/log/ceph/ceph-client.admin.39400.log 2024-03-29T10:57:10.818 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39244.log: /var/log/ceph/ceph-client.admin.41175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41721.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39400.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41175.log.gz 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41721.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.39400.log.gz -5 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41682.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41721.log.gz 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40025.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41682.log.gz 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39595.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40142.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39635.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41565.log 2024-03-29T10:57:10.819 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz/var/log/ceph/ceph-client.admin.40142.log: 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41565.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40142.log.gz 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40337.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41565.log.gz 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.3.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39439.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39713.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40337.log: /var/log/ceph/ceph-client.admin.39439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40337.log.gz 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39713.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39439.log.gz 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39478.log 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38657.log: gzip 85.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39713.log.gz -- 2024-03-29T10:57:10.820 INFO:teuthology.orchestra.run.smithi064.stderr: /var/log/ceph/ceph-client.admin.40298.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40941.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39478.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40941.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40298.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41214.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39010.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41604.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40941.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39010.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41214.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39127.log 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39010.log.gz 2024-03-29T10:57:10.821 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38618.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39127.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39127.log.gz 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38893.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40610.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28991.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38893.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41409.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.28991.log: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40823.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38893.log.gz 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28991.log.gz 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39674.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40823.log: -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:57:10.822 INFO:teuthology.orchestra.run.smithi064.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41409.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40823.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.39674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40784.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38854.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39674.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:gzip 86.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40784.log: /var/log/ceph/ceph-client.admin.40064.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40784.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38462.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39049.log 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39556.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-03-29T10:57:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39049.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: 85.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41877.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39049.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41370.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41877.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39791.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41799.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41331.log: /var/log/ceph/ceph-client.admin.39791.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39205.log 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39791.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41799.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41799.log.gz 2024-03-29T10:57:10.824 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40532.log: 85.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38737.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39205.log.gz 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39088.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41643.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40902.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38737.log.gz 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41643.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39088.log.gz 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40902.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41643.log.gz 2024-03-29T10:57:10.825 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40902.log.gz 2024-03-29T10:57:10.831 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:57:10.831 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-03-29T10:57:10.831 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34529.log 2024-03-29T10:57:10.831 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34609.log 2024-03-29T10:57:10.831 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34529.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:57:10.831 INFO:teuthology.orchestra.run.smithi174.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34609.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34529.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34609.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34409.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34569.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34409.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34649.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34409.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34449.log 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mgr.x.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34649.log.gz 2024-03-29T10:57:10.832 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34489.log 2024-03-29T10:57:10.833 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34449.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:57:10.833 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.34489.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:57:10.833 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34449.log.gz 2024-03-29T10:57:10.833 INFO:teuthology.orchestra.run.smithi174.stderr: 87.7%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.34489.log.gz 2024-03-29T10:57:10.833 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:57:10.877 INFO:teuthology.orchestra.run.smithi174.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:57:22.616 INFO:teuthology.orchestra.run.smithi174.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:57:22.616 INFO:teuthology.orchestra.run.smithi174.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:57:22.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:57:22.753 INFO:teuthology.orchestra.run.smithi174.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:57:26.390 INFO:teuthology.orchestra.run.smithi174.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:57:26.958 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:57:28.219 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:57:28.303 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:57:56.701 INFO:teuthology.orchestra.run.smithi064.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:58:04.149 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:58:04.150 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-29T10:58:04.151 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m53.390s 2024-03-29T10:58:04.151 INFO:teuthology.orchestra.run.smithi064.stderr:user 2m36.941s 2024-03-29T10:58:04.151 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m5.050s 2024-03-29T10:58:22.146 INFO:teuthology.orchestra.run.smithi174.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:58:22.148 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-29T10:58:22.148 INFO:teuthology.orchestra.run.smithi174.stderr:real 1m11.349s 2024-03-29T10:58:22.148 INFO:teuthology.orchestra.run.smithi174.stderr:user 1m57.821s 2024-03-29T10:58:22.148 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m4.422s 2024-03-29T10:58:22.148 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:58:22.149 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/remote/smithi064/log 2024-03-29T10:58:22.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:58:44.089 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/remote/smithi174/log 2024-03-29T10:58:44.091 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:59:02.428 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:59:02.635 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fd194e18ad9347e1be15037fc4ec11cd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:59:02.639 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:59:02.648 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:59:02.656 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:59:02.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:59:02.670 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:59:02.747 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:59:02.756 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:59:02.757 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:59:02.759 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:59:02.773 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:59:02.775 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:59:02.777 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:59:02.778 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-29T10:59:02.778 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +429us[ +430us] +/- 76ms 2024-03-29T10:59:02.778 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -184us[ -183us] +/- 56ms 2024-03-29T10:59:02.778 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -200us[ -199us] +/- 87ms 2024-03-29T10:59:02.778 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:59:02.780 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:59:02.780 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-29T10:59:02.780 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +416us[ +412us] +/- 77ms 2024-03-29T10:59:02.781 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -186us[ -190us] +/- 56ms 2024-03-29T10:59:02.781 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -217us[ -220us] +/- 87ms 2024-03-29T10:59:02.781 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:59:02.781 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:59:02.790 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:59:02.790 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:59:02.798 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:59:02.891 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:59:02.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:59:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:59:02.991 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:59:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:59:03.022 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:59:03.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:59:03.039 INFO:teuthology.task.internal:Duration was 1212.526855 seconds 2024-03-29T10:59:03.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:59:03.048 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:59:03.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:59:03.050 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:59:03.093 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:59:03.106 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:59:03.501 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:59:03.502 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-03-29T10:59:03.502 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:59:03.523 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-03-29T10:59:03.523 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:59:03.555 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:59:03.556 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:59:03.565 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:59:03.625 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:59:03.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:59:03.636 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:59:03.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:59:03.773 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:59:03.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:59:03.800 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:59:03.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:59:03.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:59:03.842 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:59:03.865 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-03-29T10:59:03.892 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-03-29T10:59:03.931 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:59:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:59:03.945 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:59:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:59:03.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:59:03.997 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:59:03.998 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/remote/smithi064 2024-03-29T10:59:03.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:59:04.038 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655/remote/smithi174 2024-03-29T10:59:04.039 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:59:04.074 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:59:04.074 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:59:04.076 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:59:04.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:59:04.135 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:59:04.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:59:04.145 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:59:04.146 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:59:04.149 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:59 /home/ubuntu/cephtest 2024-03-29T10:59:04.178 INFO:teuthology.orchestra.run.smithi174.stdout: 265945 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:59 /home/ubuntu/cephtest 2024-03-29T10:59:04.179 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:59:04.189 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:59:04.208 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:59:04.233 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-29T10:59:04.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:59:04.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.513095', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1XYCHPDgbbRvcQkzvM0nSiBLU6dxAKguv4WJ+e8cGFf0JzLEoLa3QcVarrIPCZovi3uCVThv/ja51YOi9SH7k='} 2024-03-29T10:59:04.264 DEBUG:teuthology.nuke:shortname: smithi174 2024-03-29T10:59:04.264 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:59:04.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622655', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.511840', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJI+GBK6tFBCw/9r9EfQ2evtADR04/h6CTSao5wyoWR+iyOenbhJP8D8WKEC+5hpebwKZf8C1TUhmL9pYHsv20c='} 2024-03-29T10:59:04.315 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-29T10:59:04.315 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:59:04.340 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-29T10:59:04.341 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:04.357 INFO:teuthology.orchestra.console.smithi174:Power off 2024-03-29T10:59:04.357 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:59:04.367 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-29T10:59:04.379 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-03-29T10:59:04.380 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:04.403 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-29T10:59:08.368 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:08.404 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:08.480 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-29T10:59:08.515 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-29T10:59:12.482 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:12.516 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:12.593 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-29T10:59:12.625 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-03-29T10:59:12.726 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-03-29T10:59:16.598 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:59:16.709 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-29T10:59:16.810 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-29T10:59:16.914 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1212.5268545150757 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fd194e18ad9347e1be15037fc4ec11cd status: fail success: false 2024-03-29T10:59:16.915 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:59:17.004 INFO:teuthology.run:FAIL